Rethink Monitoring

                 What Just Happened?
Leveraging Puppet & Boundary to Manage Dynamic Environments

               Molly Stamos, Director of Products
Boundary: Highly Distributed Platform delivering SaaS

•  Boundary is a SaaS application that monitors application performance by analyzing
    application communication over the network"
•  Platform runs on Ubuntu Linux with a mix of off the shelf and home grown components "
•  Continuous deployment shop - code rolled into production as its ready "


"
Boundary: The Team

•  “devops” shop – but two primary Ops guys managing about 200 nodes and overseeing
    all automation "
         Scott Smith             Joe Williams



"


"
•  Everyone participates in operations – everyone is on call rotation (and gets the joy of
    2am wake up calls)"
•  Environment is hosted – Provider racks & stacks – Joe & Scott do everything else"
An Example
(but first a quick look at Boundary)
Configuration Change in Production

•    Kobayashi is our historical data store "
     •    9 node cluster "
     •    Riak under the covers"
     •    Provides historical feed of data for Boundary dashboards (beyond streaming system)"


•    Configuration change needed to alleviate feared future performance problems"
     •    In Riak, must perform a “rolling upgrade”"
          •    Push config change to each node one at a time, restart services, wait for cluster to reach steady state"
     •    Unsure how cluster behavior would change "
Using Boundary to Assess Cluster Health

"
Unexpected Benefit: Evenly Distributed Writes Across the Cluster

      BEFORE"                               AFTER"
Application Latency (App RTT) Before the Change


•  Node Data06 is 2x
  slower than the rest
  of the nodes"
Application Latency (App RTT) After the Change


•  App RTT are tightly grouped
  around 4-5 ms"
Boundary & Puppet
Using the Puppet Module
Download and Install the Boundary Module

•  Fairly Straightforward:"
  •  sudo puppet module search boundary
  •  sudo puppet module install puppetlabs-boundary

 Make sure you get this one: puppetlabs-boundary (NOT the bprobe one) !


•  **Some configuration required**"
  •  Boundary::params class – set your API Key and your OrgID "
     •  Located in /etc/puppetlabs/puppet/boundary/manifests/params.pp    "


  •  Boundary.yaml file – set your API Key and OrgID "
     •  Located in /etc/puppetlabs/puppet/boundary/"
Where to get your API Key & OrgID in Boundary




•  Go to Organization -> Settings"
•  API Key & OrgID"
  •  API Key is like your password – keep it secret"
Using Puppet to Deploy the Boundary Meter
Get Boundary Now

•  Boundary: http://boundary.com/signup/"
  •  Literally takes less than 2 minutes to deploy"
     •  When was the last time a monitoring tool took less than 2 minutes to set up?"

  •  Free to use up to 2GB per day "



•  How do I get more information? "
  •  Website: www.boundary.com"
  •  Email: service@boundary.com"
Rethink Monitoring

Boundary for puppet @ puppet conf2012

  • 1.
    Rethink Monitoring What Just Happened? Leveraging Puppet & Boundary to Manage Dynamic Environments Molly Stamos, Director of Products
  • 2.
    Boundary: Highly DistributedPlatform delivering SaaS •  Boundary is a SaaS application that monitors application performance by analyzing application communication over the network" •  Platform runs on Ubuntu Linux with a mix of off the shelf and home grown components " •  Continuous deployment shop - code rolled into production as its ready " "
  • 3.
    Boundary: The Team • “devops” shop – but two primary Ops guys managing about 200 nodes and overseeing all automation " Scott Smith Joe Williams " " •  Everyone participates in operations – everyone is on call rotation (and gets the joy of 2am wake up calls)" •  Environment is hosted – Provider racks & stacks – Joe & Scott do everything else"
  • 4.
    An Example (but firsta quick look at Boundary)
  • 5.
    Configuration Change inProduction •  Kobayashi is our historical data store " •  9 node cluster " •  Riak under the covers" •  Provides historical feed of data for Boundary dashboards (beyond streaming system)" •  Configuration change needed to alleviate feared future performance problems" •  In Riak, must perform a “rolling upgrade”" •  Push config change to each node one at a time, restart services, wait for cluster to reach steady state" •  Unsure how cluster behavior would change "
  • 6.
    Using Boundary toAssess Cluster Health "
  • 7.
    Unexpected Benefit: EvenlyDistributed Writes Across the Cluster BEFORE" AFTER"
  • 8.
    Application Latency (AppRTT) Before the Change •  Node Data06 is 2x slower than the rest of the nodes"
  • 9.
    Application Latency (AppRTT) After the Change •  App RTT are tightly grouped around 4-5 ms"
  • 10.
    Boundary & Puppet Usingthe Puppet Module
  • 11.
    Download and Installthe Boundary Module •  Fairly Straightforward:" •  sudo puppet module search boundary •  sudo puppet module install puppetlabs-boundary Make sure you get this one: puppetlabs-boundary (NOT the bprobe one) ! •  **Some configuration required**" •  Boundary::params class – set your API Key and your OrgID " •  Located in /etc/puppetlabs/puppet/boundary/manifests/params.pp " •  Boundary.yaml file – set your API Key and OrgID " •  Located in /etc/puppetlabs/puppet/boundary/"
  • 12.
    Where to getyour API Key & OrgID in Boundary •  Go to Organization -> Settings" •  API Key & OrgID" •  API Key is like your password – keep it secret"
  • 13.
    Using Puppet toDeploy the Boundary Meter
  • 14.
    Get Boundary Now • Boundary: http://boundary.com/signup/" •  Literally takes less than 2 minutes to deploy" •  When was the last time a monitoring tool took less than 2 minutes to set up?" •  Free to use up to 2GB per day " •  How do I get more information? " •  Website: www.boundary.com" •  Email: service@boundary.com"
  • 15.