Lessons Learned at Wakoopa

1,610 views
1,548 views

Published on

The talk I gave at RubyEnRails 2008

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,610
On SlideShare
0
From Embeds
0
Number of Embeds
79
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Lessons Learned at Wakoopa

  1. 1. Lessons Learned while building Wakoopa
  2. 2. Menno van der Sman Developer at Wakoopa since May 2007
  3. 3. Wakoopa is about software
  4. 4. PC Mac
  5. 5. 200 million
  6. 6. Does it scale?
  7. 7. Scaling the front is easy
  8. 8. Scaling the back is harder
  9. 9. Limits spawn creativity
  10. 10. One server at Railsmachine sliced into staging, production and db MySQL 4.1 Apache + Mongrel
  11. 11. Background queue
  12. 12. Lots of tools already BackgroundRB, DelayedJob, Daemons etc
  13. 13. Rolling our own database based using script/runner processing multiple items in one loop one cache update per loop
  14. 14. Smart queries UPDATE developers SET active_seconds = active_seconds + ... INSERT INTO hourly_usage (...) VALUES (...) ON DUPLICATE KEY UPDATE active_seconds = active_seconds + ..., idle_seconds = idle_seconds + ... Allows for concurrency
  15. 15. Size = rowsize x rows Make rowsize smaller
  16. 16. sometimes `id` shouldn’t be the Primary Key
  17. 17. Enter composite_primary_keys
  18. 18. Overrides default PK class HourlyUsage < ActiveRecord::Base set_primary_keys :user_id, :software_id, :used_at ... end Original idea by Dr Nic Enhanced for Rail 2.1 by Darrin Holst http://compositekeys.rubyforge.org/
  19. 19. Archive old data that’s not frequently used
  20. 20. HourlyUsage to HourlyUsageStorage SELECT ... FROM hourly_usage WHERE used_at < '...' INTO OUTFILE '/path/to/file' LOAD DATA INFILE '/path/to/file' REPLACE INTO TABLE hourly_usage_storage avoids long locks
  21. 21. Upcoming improvements
  22. 22. MySQL 5.1 Partitioning CREATE TABLE hourly_usage_storage ( ... ) PARTITION BY RANGE (software_id) ( PARTITION p0 VALUES LESS THAN (...), PARTITION p1 VALUES LESS THAN (...), PARTITION p2 VALUES LESS THAN (...), PARTITION p3 VALUES LESS THAN (...), .... PARTITION pn VALUES LESS THAN MAXVALUE ); Makes ‘WHERE software_id = ...’ very fast Available in MySQL 5.1
  23. 23. And many more plans Designing a new infrastructure MySQL 5.1 Memcached Sphinx Rails 2.1 Amazon EC2
  24. 24. Announcing a new blog Inside Wakoopa http://inside.wakoopa.com
  25. 25. Questions? or suggestions? Looking for C++ and frontend awesomeness http://wakoopa.com/jobs

×