How Edmodo Uses Splunk App for AWS

Jack Murgia
NetOps Architect
jack@edmodo.com
Edmodo is the largest social learning network
in the world

28,000,000+
TOTAL USERS AROUND THE WORLD
The Edmodo user base has grown REALLY
FAST and so has our AWS usage.
• New Features
20M+ users

18,000,000

▸ Require new backend clusters
• More Employees

14,000,000

▸ Launch more development
servers

20M+ Registered Users

• New Investors
▸ Want assurances we spend
money wisely

10,000,000

Investors

6,000,000
4.8M users
1.1M users

2,000,000

225K users

Dec 09

Dec 10

Dec 11

Dec 12
100’s of
production, development, reporting
and operations instances are
managed on Amazon Web
Services Elastic Compute Cloud.
20 employees across 5 teams can
launch EC2 instances.
We tried lots of slow, pricey, hosted apps.
None had what we REALLY Needed …
Real Time RI vs On-Demand Usage
Do we have enough RI coverage? Are our RIs being used?
Getting Usage Data In front of Teams
Making Real Time Usage Data Visible Encourages Action

Problem:
The default Splunk for AWS App configuration is
based on billing “subaccounts”, which Edmodo
does not use.
Edmodo uses tags to differentiate between cost
centers.
Use These Tags: Product, Service, Space
Configurable Metadata Available for All AWS Resources
• Product
▸ designation is derived from the
end user perspective. Where
component resources serve
overlapping Products, the
Product designation is that which
uses/requires a resource most or
the resource's reason for existing.
• Service is the primary function of
the resource. Examples are
▸ mysql

▸ apache
▸ active-directory
• Space a logical workplace
designation. Examples are:
▸ Development
▸ Production
Edit the Splunk For AWS Scripts for Tags
get_instances.py and get_ops.py
Create Lookup Tables For The Tags
Simple .csv files are referenced by dashboard view files
Personalize Your Dashboard Views
The Dashboard XML format is easy to customize
Drill Down Into Usage Data by Tags
Teams Can Now Evaluate Usage For The Services They Manage
Drill Down Into Usage Data by Tags
Teams Can Now Evaluate Usage For The Services They Manage
Drill Down Into Usage Data by Tags
Teams Can Now Evaluate Usage For The Services They Manage
Download Tag Customizations

http://public.edim.com.s3.amazonaws.com/splunk-aws-app-for-tags.tgz

How Edmodo Uses Splunk For Real-Time Tag-Based Reporting of AWS Billing and Usage

  • 1.
    How Edmodo UsesSplunk App for AWS Jack Murgia NetOps Architect jack@edmodo.com
  • 2.
    Edmodo is thelargest social learning network in the world 28,000,000+ TOTAL USERS AROUND THE WORLD
  • 3.
    The Edmodo userbase has grown REALLY FAST and so has our AWS usage. • New Features 20M+ users 18,000,000 ▸ Require new backend clusters • More Employees 14,000,000 ▸ Launch more development servers 20M+ Registered Users • New Investors ▸ Want assurances we spend money wisely 10,000,000 Investors 6,000,000 4.8M users 1.1M users 2,000,000 225K users Dec 09 Dec 10 Dec 11 Dec 12
  • 4.
    100’s of production, development,reporting and operations instances are managed on Amazon Web Services Elastic Compute Cloud. 20 employees across 5 teams can launch EC2 instances.
  • 5.
    We tried lotsof slow, pricey, hosted apps. None had what we REALLY Needed …
  • 6.
    Real Time RIvs On-Demand Usage Do we have enough RI coverage? Are our RIs being used?
  • 7.
    Getting Usage DataIn front of Teams Making Real Time Usage Data Visible Encourages Action Problem: The default Splunk for AWS App configuration is based on billing “subaccounts”, which Edmodo does not use. Edmodo uses tags to differentiate between cost centers.
  • 8.
    Use These Tags:Product, Service, Space Configurable Metadata Available for All AWS Resources • Product ▸ designation is derived from the end user perspective. Where component resources serve overlapping Products, the Product designation is that which uses/requires a resource most or the resource's reason for existing. • Service is the primary function of the resource. Examples are ▸ mysql ▸ apache ▸ active-directory • Space a logical workplace designation. Examples are: ▸ Development ▸ Production
  • 9.
    Edit the SplunkFor AWS Scripts for Tags get_instances.py and get_ops.py
  • 10.
    Create Lookup TablesFor The Tags Simple .csv files are referenced by dashboard view files
  • 11.
    Personalize Your DashboardViews The Dashboard XML format is easy to customize
  • 12.
    Drill Down IntoUsage Data by Tags Teams Can Now Evaluate Usage For The Services They Manage
  • 13.
    Drill Down IntoUsage Data by Tags Teams Can Now Evaluate Usage For The Services They Manage
  • 14.
    Drill Down IntoUsage Data by Tags Teams Can Now Evaluate Usage For The Services They Manage
  • 15.

Editor's Notes

  • #4 The power of Edmodo has now fuelled record-breaking growth It's no longer about bringing teachers into the 21st century, but how you can leverage the network effort. Up 240% in user sign-ups since last school year (Sept 2011 data)1 in 7 of all teachers use EdmodoIn 85% of top US school districts1 in 7 teacher calculation derivation:- Assume 80% of all 12M are US- 1:10 ratio of teachers to students- 7.2M teachers K-12 in US according to US 2010 Census
  • #5 95%of schools are connected to broadband600 K-12 Districts provide tablets to students55% of students access the Internet at home through mobile devices76% of teachers stream or download videos in classiPad sales in schools have doubled since last year