Scaling Ruby from the
      Inside Out
    Ezra Zygmuntowicz
Does Rails Scale?
Does Rails Scale?

Of course it does...
Does Ruby Scale?
Does Ruby Scale?

Of course it does...
Should you use Rails or
        Merb?
Should you use Rails or
        Merb?

         YES!
Are you sick and tired
of endlessly debating
      this crap?
Are you sick and tired
of endlessly debating
      this crap?
 Of course you are...
Me too, let’s talk about
  something more
     interesting...
Scaling Ruby from the
      Inside Out
What should we talk
   about then?
What should we talk
   about then?

The weather maybe?
Forecast calls for 100%
  chance of clouds...
Cloud Computing
Cloud Computing
   Platforms
 Abstract/Virtual Resources
            CPU
          Memory
          Storage
        API Ac...
Cloudy Weather
• Amazon EC2     • 3terra
• Engine Yard    • Google App Engine
• Joyent         • Project Caroline
• Mosso ...
There will be *many*
  cloud computing
platforms in the near
       future
Many virtual machines
     to manage
SSH doesn’t scale...
SSH doesn’t scale...
Takes time to setup/teardown
         connections
SSH doesn’t scale...
 Takes time to setup/teardown
          connections

Inherently push only technology
Capistrano is an awesome tool
Capistrano is an awesome tool
  but running commands on
   thousands of machines in
  parallel is pushing it further
     ...
How will we manage
  the complexity?
How will we manage
  the complexity?

  It’s all about the
    distribution...
We’ve got to think big now!
 New York, Chicago, LA
Introducing Vertebra
 Next Generation Cloud
 Computing/Automation
      Framework
Enter XMPP
Extensible Messaging and Presence Protocol
Enter XMPP
Extensible Messaging and Presence Protocol




           Extensible
                I like that
Enter XMPP
      Extensible Messaging and Presence Protocol




                 Messaging
We want to send messages to our...
Enter XMPP
     Extensible Messaging and Presence Protocol




                 Presence
We want to know when our machines...
XMPP is a realtime
messaging protocol built for
     IM/chat, great for
 communication between
 thousands of people, why
 ...
Summoning Slices via XMPP
Ejabberd

 Horizontally scalable/federated erlang XMPP/Jabber server

High performance even with tens of thousands of conn...
Ruby based agents run on
every slice/instance/node
      in the cloud
Presence let's us know when
 agents come online or go
           offline
Security/Discovery/
 Introduction Framework
Keeps Agents in Touch with
       Each Other.
Distributed Hierarchal Key-
   Value Storage System
    Queried via XMPP
State Machine/Workflow
Engine Coordinates Multiple
      Agents/Humans
Vertebra
CREST
CREST
Cloud-REST
CREST
   Cloud-REST
Operations on Resources
      in the cloud
operation:list
/cluster/ey05
  /node/42
     /xen
Resource Discovery
Resource Discovery

Who provides this set of resources?
Resource Discovery

Who provides this set of resources?
Map/Reduce
Scatter/Gather Dispatch
Map/Reduce
 Scatter/Gather Dispatch
Discover
Map/Reduce
 Scatter/Gather Dispatch
Discover   Scatter
Map/Reduce
 Scatter/Gather Dispatch
Discover   Scatter   Gather
quick demo
Not only server
 automation
Ejabberd + Comet + Jsjac
           ==
 Realtime Web Platform
Many Ideas to
Experiment With
XMPP > HTTP
Watch this space...
One more thing...
One more thing...

             Introducing:
          Engine Yard Express
http://express.engineyard.com/system/ey_express...
Engine Yard Express




http://express.engineyard.com/system/ey_express_v0.1.tar.bz2
Questions?
Vertebra
Upcoming SlideShare
Loading in...5
×

Vertebra

27,167

Published on

This is my presentation from RailsConf 2008 introducing Vertebra, our new cloud computing application platform

Published in: Technology

Vertebra

  1. 1. Scaling Ruby from the Inside Out Ezra Zygmuntowicz
  2. 2. Does Rails Scale?
  3. 3. Does Rails Scale? Of course it does...
  4. 4. Does Ruby Scale?
  5. 5. Does Ruby Scale? Of course it does...
  6. 6. Should you use Rails or Merb?
  7. 7. Should you use Rails or Merb? YES!
  8. 8. Are you sick and tired of endlessly debating this crap?
  9. 9. Are you sick and tired of endlessly debating this crap? Of course you are...
  10. 10. Me too, let’s talk about something more interesting...
  11. 11. Scaling Ruby from the Inside Out
  12. 12. What should we talk about then?
  13. 13. What should we talk about then? The weather maybe?
  14. 14. Forecast calls for 100% chance of clouds...
  15. 15. Cloud Computing
  16. 16. Cloud Computing Platforms Abstract/Virtual Resources CPU Memory Storage API Access
  17. 17. Cloudy Weather • Amazon EC2 • 3terra • Engine Yard • Google App Engine • Joyent • Project Caroline • Mosso • GoGrid This is only the beginning...
  18. 18. There will be *many* cloud computing platforms in the near future
  19. 19. Many virtual machines to manage
  20. 20. SSH doesn’t scale...
  21. 21. SSH doesn’t scale... Takes time to setup/teardown connections
  22. 22. SSH doesn’t scale... Takes time to setup/teardown connections Inherently push only technology
  23. 23. Capistrano is an awesome tool
  24. 24. Capistrano is an awesome tool but running commands on thousands of machines in parallel is pushing it further then it wants to go
  25. 25. How will we manage the complexity?
  26. 26. How will we manage the complexity? It’s all about the distribution...
  27. 27. We’ve got to think big now! New York, Chicago, LA
  28. 28. Introducing Vertebra Next Generation Cloud Computing/Automation Framework
  29. 29. Enter XMPP Extensible Messaging and Presence Protocol
  30. 30. Enter XMPP Extensible Messaging and Presence Protocol Extensible I like that
  31. 31. Enter XMPP Extensible Messaging and Presence Protocol Messaging We want to send messages to our thousands of machines
  32. 32. Enter XMPP Extensible Messaging and Presence Protocol Presence We want to know when our machines are online or not and act accordingly
  33. 33. XMPP is a realtime messaging protocol built for IM/chat, great for communication between thousands of people, why not machines?
  34. 34. Summoning Slices via XMPP
  35. 35. Ejabberd Horizontally scalable/federated erlang XMPP/Jabber server High performance even with tens of thousands of connections Forms the backbone of the system
  36. 36. Ruby based agents run on every slice/instance/node in the cloud
  37. 37. Presence let's us know when agents come online or go offline
  38. 38. Security/Discovery/ Introduction Framework Keeps Agents in Touch with Each Other.
  39. 39. Distributed Hierarchal Key- Value Storage System Queried via XMPP
  40. 40. State Machine/Workflow Engine Coordinates Multiple Agents/Humans
  41. 41. Vertebra
  42. 42. CREST
  43. 43. CREST Cloud-REST
  44. 44. CREST Cloud-REST Operations on Resources in the cloud
  45. 45. operation:list /cluster/ey05 /node/42 /xen
  46. 46. Resource Discovery
  47. 47. Resource Discovery Who provides this set of resources?
  48. 48. Resource Discovery Who provides this set of resources?
  49. 49. Map/Reduce Scatter/Gather Dispatch
  50. 50. Map/Reduce Scatter/Gather Dispatch Discover
  51. 51. Map/Reduce Scatter/Gather Dispatch Discover Scatter
  52. 52. Map/Reduce Scatter/Gather Dispatch Discover Scatter Gather
  53. 53. quick demo
  54. 54. Not only server automation
  55. 55. Ejabberd + Comet + Jsjac == Realtime Web Platform
  56. 56. Many Ideas to Experiment With
  57. 57. XMPP > HTTP
  58. 58. Watch this space...
  59. 59. One more thing...
  60. 60. One more thing... Introducing: Engine Yard Express http://express.engineyard.com/system/ey_express_v0.1.tar.bz2
  61. 61. Engine Yard Express http://express.engineyard.com/system/ey_express_v0.1.tar.bz2
  62. 62. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×