Can railsscale

  • 165 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
165
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Can Rails (finally) scale? Experience at Ask.fm Tuesday, September 24, 13
  • 2. CanRailsScale.com Tuesday, September 24, 13
  • 3. Ask.fm • 80M registered users • 200M monthly unique users • 400M page-views a day • ~1M users online Tuesday, September 24, 13
  • 4. Ask.fm • 600 questions a second • 20k requests / second • 50k likes / minute Tuesday, September 24, 13
  • 5. Ask.fm • ~300 ruby on rails nodes • 16 Cores • 16Gb RAM • 24 passengers Tuesday, September 24, 13
  • 6. Team 10 developers 2 Ruby developers Tuesday, September 24, 13
  • 7. Proposed topics • Persistence Layer • Application Layer • Infrastructure • Code scalability • Rails Tools Scalability • Misc Tuesday, September 24, 13
  • 8. Proposed topics • Persistence Layer • Application Layer • Infrastructure • Code scalability • Rails Tools Scalability • Misc Tuesday, September 24, 13
  • 9. Persistence Layer Tuesday, September 24, 13
  • 10. Persistence Layer Issues • Connection pooling • Datasource switching • High-Availability • Redis Tuesday, September 24, 13
  • 11. Persistence Layer Issues • Connection pooling • Datasource switching • High-Availability • Redis Tuesday, September 24, 13
  • 12. Connection Pool Tuesday, September 24, 13
  • 13. Connection Pool Tuesday, September 24, 13
  • 14. Connection Pool Tuesday, September 24, 13
  • 15. Connection Pool Tuesday, September 24, 13
  • 16. Persistence Layer Issues • Connection pooling • Datasource switching • High-Availability • Redis Tuesday, September 24, 13
  • 17. Datasource switching Database ActiveRecord db_charmer* Data Access Logic Business Logic * https://github.com/kovyrin/db-charmer Tuesday, September 24, 13
  • 18. Example production: bar: adapter: mysql username: blah host: blah.local database: blah foo: adapter: mysql username: foo host: foo.local database: foo class Foo < ActiveRecord::Base db_magic :connection => :foo end class Bar < ActiveRecord::Base db_magic :connection => :bar, :slave => :bar_slave end Tuesday, September 24, 13
  • 19. Persistence Layer Issues • Connection pooling • Datasource switching • High-Availability • Redis Tuesday, September 24, 13
  • 20. High-Availability Database ActiveRecord db_charmer* Data Access Logic Business Logic * https://github.com/kovyrin/db-charmer Tuesday, September 24, 13
  • 21. • Multiple datasources • Master - Slave • Sharding • High-Availability • Read & Write • Read • None High-Availability Tuesday, September 24, 13
  • 22. Sharding types • Modulo function • Data-dictionary Tuesday, September 24, 13
  • 23. Example (1) question_shard_244_247: <<: *question_shard_host_base_srvb71 database: shard244_247 shards: 244..247 question_shard_248_251: <<: *question_shard_host_base_srvb71 database: shard248_251 shards: 248..251 Tuesday, September 24, 13
  • 24. Example (2) class User < ActiveRecord::Base def questions(access_mode=:read) on_question_shard(self, access_mode) end end user.questions().answered.latest user.questions(:read_write).build({...}) Tuesday, September 24, 13
  • 25. Example (3) def on_question_shard(access_mode=:read_write, &block) accessor = shard_accessor(access_mode) question_data_source.shard_for(accessor, nil, &block) end def shard_for_key(accessor) raise ShardAccessError, 'accessor missing' unless accessor raise ShardAccessError, 'access denied' unless @shard_access.shard_accessible?(accessor.key, accessor.access_mode) shard_connection_config( data_source_name(accessor.key) ) end Tuesday, September 24, 13
  • 26. Persistence Layer Issues • Connection pooling • Datasource switching • High-Availability • Redis Tuesday, September 24, 13
  • 27. Redis Sharding • ShardMap • HashRing • DataDictionary Tuesday, September 24, 13
  • 28. Proposed topics • Persistence Layer • Application Layer • Infrastructure • Code scalability • Rails Tools Scalability • Misc Tuesday, September 24, 13
  • 29. Application Layer • Ruby threads • Configuration • Queue • Uploads • Production Log • Statistics Tuesday, September 24, 13
  • 30. Application Layer • Ruby threads • Configuration • Queue • Uploads • Production Log • Statistics Tuesday, September 24, 13
  • 31. Ruby threads • Multithreading in Ruby sucks :) Tuesday, September 24, 13
  • 32. Ruby threads • Parallel DB calls • Third-party services • Any service :) Tuesday, September 24, 13
  • 33. Application Layer • Ruby threads • Configuration • Queue • Uploads • Production Log • Statistics Tuesday, September 24, 13
  • 34. Configuration • database.yml • email.yml • redis.yml • ... • system.yml • ... Tuesday, September 24, 13
  • 35. Centralized configuration Tuesday, September 24, 13
  • 36. Centralized configuration if defined?(PhusionPassenger) PhusionPassenger.on_event(:starting_worker_process) do |forked| if forked Rlt::Configuration::UpdateWatcher::start(askfm_config.store) else end end else Rlt::Configuration::UpdateWatcher::start(askfm_config.store) end Tuesday, September 24, 13
  • 37. Application Layer • Ruby threads • Configuration • Queue • Uploads • Production Log • Statistics Tuesday, September 24, 13
  • 38. Queue • Resque Tuesday, September 24, 13
  • 39. Application Layer • Ruby threads • Configuration • Queue • Uploads • Production Log • Statistics Tuesday, September 24, 13
  • 40. Upload • Goliath* application (non-blocking ruby web server) • 4 servers • 2.5M Uploads a day * https://github.com/postrank-labs/goliath/ Tuesday, September 24, 13
  • 41. Application Layer • Ruby threads • Configuration • Queue • Uploads • Production Log • Statistics Tuesday, September 24, 13
  • 42. Graylog Tuesday, September 24, 13
  • 43. Application Layer • Ruby threads • Configuration • Queue • Uploads • Production Log • Statistics Tuesday, September 24, 13
  • 44. Statistics • Business metrics • Application metrics • Performance metrics • Errors Tuesday, September 24, 13
  • 45. Statistics Tuesday, September 24, 13
  • 46. Statistics Tuesday, September 24, 13
  • 47. Proposed topics • Persistence Layer • Application Layer • Infrastructure • Code scalability • Rails Tools Scalability • Misc Tuesday, September 24, 13
  • 48. Infrastructure • Memory leaks • Timeout Tuesday, September 24, 13
  • 49. Proposed topics • Persistence Layer • Application Layer • Infrastructure • Code scalability • Rails Tools Scalability • Misc Tuesday, September 24, 13
  • 50. Code scalability • MVC • ActiveRecord • Tests • DB Migrations Tuesday, September 24, 13
  • 51. Proposed topics • Persistence Layer • Application Layer • Infrastructure • Code scalability • Rails Tools Scalability • Misc Tuesday, September 24, 13
  • 52. Rails tools scalability • NewRelic • Exceptional.io • Google Analytics • Capistrano Tuesday, September 24, 13
  • 53. Proposed topics • Persistence Layer • Application Layer • Infrastructure • Code scalability • Rails Tools Scalability • Misc Tuesday, September 24, 13
  • 54. GC Tunning Tuesday, September 24, 13
  • 55. Ruby 2.0 Tuesday, September 24, 13
  • 56. jRuby Tuesday, September 24, 13
  • 57. Event Machine Tuesday, September 24, 13