Managing RightScale on RightScale

931 views

Published on

Rafael Saavedra, VP Engineering at RightScale, led this session at the RightScale User Conference 2010 in Santa Clara.

Session Abstract: Just like our customers, RightScale runs in the cloud and requires the best platform to automate operations. As such, RightScale uses RightScale to manage RightScale. Our complete infrastructure – development, testing, staging, and production – consists of servers that are configured, launched and managed by the RightScale Platform. In this talk, we'll present insights into how our different systems are set up and managed through the RightScale dashboard, how we organize the different production deployments, how we roll out major and minor upgrades to our infrastructure, and what best practices we follow during normal and emergency operations.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
931
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
30
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • RightScale'sServerTemplates allow you to capture best practices for provisioning and automating cloud infrastructure.  In this breakout session, we will explore how you can leverage the RightScale platform to share ServerTemplates with others.  Specifically, we'll walk through the steps to share and update ServerTemplates across your organization.  We'll also show you how to publish ServerTemplates publicly for the whole world to use.  This topic is best for: IT members who are responsible for maintaining server configurations within the organization, developers who would like to share work product within their group or ISVs wishing to reach cloud users by publishing through RightScale.
  • Managing RightScale on RightScale

    1. 1. Managing RightScale on RightScale<br />Rafael H. Saavedra<br />VP of Engineering<br />
    2. 2. Topics<br />RightScale managed by RightScale<br />Meta, production, staging & development<br />An overview of the production system<br />QuisCustodietIpsosCustodes<br />Deploying RightScale – best practices <br />What we love about using RightScale <br />Features that are difficult to use<br />
    3. 3. RightScale: Cloud Management Platform<br />RightScale Production<br />Customer A<br />Customer D<br />Customer B<br />Customer C<br />
    4. 4. RightScale: Cloud Management Platform<br />RightScale Meta Production<br />RightScale<br />Production<br />RightScale<br />Staging<br />RightScale<br />Development<br />Customer A<br />Customer D<br />RightScale<br />Development<br />
    5. 5. A multitude of RightScale systems<br />Meta Production currently lives outside the cloud <br />Use only to manage the production system<br />Only RightScale ops accounts<br />Production: my.rightscale.com<br />Reaching 200 servers, a large fraction in EC2 us-east<br />Servers in every cloud to achieve high availability<br />Servers allocated in well defined availability zones<br />A few staging systems used for integration and QA<br />Ad hoc systems for performance testing, demos, betas<br />Many development systems with simplified configurations<br />A development system at the click of a button<br />
    6. 6. Significant increase in cloud usage<br />
    7. 7. Some interesting RightScale numbers<br />1.65M servers launched by RightScale<br />RightScale continuously monitors more than 60k servers<br />Every day at RightScale:<br />2,000 array resize actions are executed<br />35,000 alert escalations are triggered<br />20,000 escalation emails are sent to users<br />9.0TB of monitoring data is exchange with our servers<br />1.6TB of logging data is sent to our servers<br />
    8. 8. RightScale production – simplified<br />others<br />Main App<br />Front Ends<br />logging<br />API<br />dashboard<br />databases<br />daemons<br />DB Master<br />monitoring<br />DB Slave<br />mirrors<br />
    9. 9. What is that our users do?<br />Dashboard, API, monitoring graphs & event notifications<br />Most of the requests are monitoring updates 85% (70%)<br />Dashboard and API represent 7% of requests but 26% of traffic<br />
    10. 10. We eat our own dog food<br />Production servers organized into independent deployments<br />Core servers: frontends, core/api servers, databases, daemons<br />
    11. 11. We eat our own dog food<br />Extensive use of security groups to isolate servers<br />ServerTemplates are maintained for each major release<br />Ability to launch exact configurations of past versions <br />
    12. 12. Monitoring, alerts & escalations<br />Monitor as much as possible, what is relevant and display it in insightful ways<br />The need to quickly detect patterns and abnormalities<br />Proactively eliminate the conditions that raise critical alerts<br />No broken windows policy<br /> APIs Cores<br />
    13. 13. QuisCustodietIpsosCustodes?*<br />The need to monitor the monitoring and alerting systems<br />Extensive use of alerts to monitor the responsiveness of all the RightScale servers<br />Instance and EBS failures gives us headaches<br />Decoupling the meta & production monitoring and alerting systems<br />* Who watches the watchmen?<br />
    14. 14. How to Monitor hundreds of servers?<br />Starting to use stacked graphs & heat maps<br />The need to quickly detect patterns and abnormalities<br />
    15. 15. Our favorite RightScale features<br />RightImages: never again the need to build custom images<br />Input inheritance: makes it easy to keep the configurations of dozens of servers in sync <br />ServerTemplates: very easy to reproduce configurations in production, staging and development<br />The Library: there is always an example of something new that can be adapted to our needs<br />Monitoring: easy to make a collectdplugins to monitor just about anything<br />
    16. 16. Our not so favorite features<br />ServerTemplate inputs: powerful but too many of them make templates difficult to use<br />Revision management: a way to go to make users aware of new revisions and version and how to update<br />The Library: checking out new resources from library is not easy<br />Alerts: they work pretty well but they are not easy to configure, in particular, custom ones<br />
    17. 17. Best practices: upgrading RightScale<br />Avoid upgrading existing servers; instead launch fresh ones with new software (fail forward)<br />Not possible on some components, e.g. monitoring servers, which are in the hundreds<br />The cost of duplicating servers is minimal <br />Old servers can take over in case something goes wrong <br />Launch additional slaves to capture recovery points<br />One slave continues to replicate in case of master failure<br />Another slave is frozen at upgrade point – can rollback by failing over<br />Don’t forget to take snapshots in case of major failure<br />
    18. 18. Upgrading RightScale: Step by Step<br />Front Ends<br />Main App<br />Main App<br />servers with new code<br />take snapshot at cutoff<br />Databases<br />DB Master<br />DB Slave<br />DB Slave<br />stop replication<br />cut access to site<br />stop all access to DB<br />servers with old code<br />
    19. 19. Upgrading RightScale: Step by Step<br />Front Ends<br />Main App<br />Main App<br />servers with new code<br />snapshot at cutoff<br />reconnect all servers<br />Databases<br />DB Master<br />DB Slave<br />DB Slave<br />stop replication<br />servers with old code<br />
    20. 20. Upgrading RightScale: Step by Step<br />Front Ends<br />Main App<br />Main App<br />servers with new code<br />reconnect all servers<br />open access to site<br />Databases<br />DB Master<br />DB Slave<br />DB Slave<br />servers with old code<br />

    ×