Monitor all the Things!

KEVIN MCGUIRE - DIRECTOR OF ENGINEERING, PLATFORM
OCTOBER 25/2013

Wednesday, November 6, 13
The evolution of monitoring

Wednesday, November 6, 13
The evolution of monitoring

Wednesday, November 6, 13
The evolution of monitoring

Wednesday, November 6, 13
The evolution of monitoring

Wednesday, November 6, 13
Your end user experience depends on more

Wednesday, November 6, 13
A lot more

Wednesday, November 6, 13
You can’t manage what you can’t monitor

Wednesday, November 6, 13
You can’t manage what you can’t monitor

Wednesday, November 6, 13
You can’t manage what you can’t monitor

Wednesday, November 6, 13
Solution?

Wednesday, November 6, 13
Solution?
MONITOR ALL THE THINGS!

http://hyperboleandahalf.blogspot.com/

Wednesday, November 6, 13
Platform gives complete visibility

Wednesday, November 6, 13
Platform gives complete visibility

Wednesday, November 6, 13
Platform gives complete visibility

Wednesday, November 6, 13
Plugins for, well, everything!
To
ob
ig
t

68 69 published plugins
29 different publishers

of

Wednesday, November 6, 13
...
We got databases

Wednesday, November 6, 13
We got yer NoSQL stores

Wednesday, November 6, 13
Yup, caching too

Wednesday, November 6, 13
Queuing, web server, load blance, ...

Wednesday, November 6, 13
And my personal favorite:

Wednesday, November 6, 13
And my personal favorite:

You go Yuri Yasiyarov, wherever you are!

Wednesday, November 6, 13
Integrated, 1st class experience

Wednesday, November 6, 13
Integrated, 1st class experience

Wednesday, November 6, 13
Integrated, 1st class experience

Wednesday, November 6, 13
Integrated, 1st class experience

Wednesday, November 6, 13
Integrated, 1st class experience

Wednesday, November 6, 13
Customized charts
MySQL plugin has MySQL’y stuff

Wednesday, November 6, 13
Install is a snap
Easy as...
1

Wednesday, November 6, 13
Install is a snap
Easy as...
1

2
[
{
"name" : "Localhost",
"host" : "localhost",
"user" : "USER_NAME_HERE",
"passwd" : "U...
Install is a snap
Easy as...
1

2

3

[

java -jar newrelic_mysql_plugin.jar
{
"name" : "Localhost",
"host" : "localhost",...
Independent runtime
Infrastructure
Varnish
Server

Memcached
Server

MySQL
Server 1
MySQL
Server 2

Externally monitored

...
SaaS
SaaS Service
Customer A

Customer B B
Customer
Customer B
Data
Customer B
Data
Data

Customer X

Customer X
license k...
Don’t see one you like?

Built 9 prior to launch,
now at 14!

Wednesday, November 6, 13

Built 6!

Built 4!

Built 3!

Bui...
Simple APIs + SDKs = Easy to write

Wednesday, November 6, 13
Simple APIs + SDKs = Easy to write
•SDKs do all the “heavy lifting”
•Java (primary), Ruby
•Runs polling cycle
•Connecting ...
Simple APIs + SDKs = Easy to write
•SDKs do all the “heavy lifting”
•Java (primary), Ruby
•Runs polling cycle
•Connecting ...
Anatomy of a plugin agent
(yes, it’s really that simple)
public	
  class	
  MySQLAgent	
  extends	
  Agent	
  {
	
  	
  pu...
Anatomy of a plugin agent
(yes, it’s really that simple)
public	
  class	
  MySQLAgent	
  extends	
  Agent	
  {
	
  	
  pu...
Anatomy of a plugin agent
(yes, it’s really that simple)
public	
  class	
  MySQLAgent	
  extends	
  Agent	
  {
	
  	
  pu...
Anatomy of a plugin agent
(yes, it’s really that simple)
public	
  class	
  MySQLAgent	
  extends	
  Agent	
  {
	
  	
  pu...
Anatomy of a plugin agent
(yes, it’s really that simple)
public	
  class	
  MySQLAgent	
  extends	
  Agent	
  {
	
  	
  pu...
Anatomy of a plugin agent
(yes, it’s really that simple)
public	
  class	
  MySQLAgent	
  extends	
  Agent	
  {
	
  	
  pu...
Anatomy of a plugin agent
(yes, it’s really that simple)
public	
  class	
  MySQLAgent	
  extends	
  Agent	
  {
	
  	
  pu...
Easy to build UIs

Wednesday, November 6, 13
Custom Dashboards
Fast and easy UI creation

Wednesday, November 6, 13
Optionally publish
Lists in Plugin Central, makes available the UIs

Wednesday, November 6, 13
Install UI = run the agent
Infrastructure
Varnish
Server

Memcached
Server

MySQL
Server 1
MySQL
Server 2

Externally moni...
Install UI = run the agent
Infrastructure
Varnish
Server

Memcached
Server

MySQL
Server 1
MySQL
Server 2

Externally moni...
“But Kevin, how much will all this cost?”

Wednesday, November 6, 13
“But Kevin, how much will all this cost?”

Free!!
Wednesday, November 6, 13
New! Plugin community reviews
Help improve plugins, give feedback

Wednesday, November 6, 13
New! Private publishing
Distribute within multiple accounts, closed Betas

Wednesday, November 6, 13
In progress: Deployment recipes
Simplified configuration and deployment across your data center

Wednesday, November 6, 13
Get involved!
✓

Review plugins

✓ Contribute to the plugins
• Many are open source
• Write one!
✓

Contribute to the SDKs...
Thank you!

Wednesday, November 6, 13
Upcoming SlideShare
Loading in …5
×

FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

844 views

Published on

Your application stack is made up of a myriad assortment of technologies, from cloud services, to the database, caching, networking, queuing and more. In order to gain a comprehensive view of your application performance, you need performance visibility into all these technologies. And, you'd like that visibility within a single tool that you already know and love.
The New Relic Platform gives you exactly that, an in-depth, real time insight into the performance of each component of your application stack. And best of all, it's free!
Find out all about the New Relic Platform and learn what it has to offer you in its wide selection of Plugins, providing insight into AWS, MySQL, Redis, MongoDB, RabbitMQ, Varnish, and many many more. Its extensible architecture means that if there isn't a plugin for the technology you want, writing one is not only possible, but easy!

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
844
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

  1. 1. Monitor all the Things! KEVIN MCGUIRE - DIRECTOR OF ENGINEERING, PLATFORM OCTOBER 25/2013 Wednesday, November 6, 13
  2. 2. The evolution of monitoring Wednesday, November 6, 13
  3. 3. The evolution of monitoring Wednesday, November 6, 13
  4. 4. The evolution of monitoring Wednesday, November 6, 13
  5. 5. The evolution of monitoring Wednesday, November 6, 13
  6. 6. Your end user experience depends on more Wednesday, November 6, 13
  7. 7. A lot more Wednesday, November 6, 13
  8. 8. You can’t manage what you can’t monitor Wednesday, November 6, 13
  9. 9. You can’t manage what you can’t monitor Wednesday, November 6, 13
  10. 10. You can’t manage what you can’t monitor Wednesday, November 6, 13
  11. 11. Solution? Wednesday, November 6, 13
  12. 12. Solution? MONITOR ALL THE THINGS! http://hyperboleandahalf.blogspot.com/ Wednesday, November 6, 13
  13. 13. Platform gives complete visibility Wednesday, November 6, 13
  14. 14. Platform gives complete visibility Wednesday, November 6, 13
  15. 15. Platform gives complete visibility Wednesday, November 6, 13
  16. 16. Plugins for, well, everything! To ob ig t 68 69 published plugins 29 different publishers of Wednesday, November 6, 13 it i n pre s en tat ion !
  17. 17. We got databases Wednesday, November 6, 13
  18. 18. We got yer NoSQL stores Wednesday, November 6, 13
  19. 19. Yup, caching too Wednesday, November 6, 13
  20. 20. Queuing, web server, load blance, ... Wednesday, November 6, 13
  21. 21. And my personal favorite: Wednesday, November 6, 13
  22. 22. And my personal favorite: You go Yuri Yasiyarov, wherever you are! Wednesday, November 6, 13
  23. 23. Integrated, 1st class experience Wednesday, November 6, 13
  24. 24. Integrated, 1st class experience Wednesday, November 6, 13
  25. 25. Integrated, 1st class experience Wednesday, November 6, 13
  26. 26. Integrated, 1st class experience Wednesday, November 6, 13
  27. 27. Integrated, 1st class experience Wednesday, November 6, 13
  28. 28. Customized charts MySQL plugin has MySQL’y stuff Wednesday, November 6, 13
  29. 29. Install is a snap Easy as... 1 Wednesday, November 6, 13
  30. 30. Install is a snap Easy as... 1 2 [ { "name" : "Localhost", "host" : "localhost", "user" : "USER_NAME_HERE", "passwd" : "USER_PASSWD_HERE" }, ] licenseKey=blahblahblah12345 Wednesday, November 6, 13
  31. 31. Install is a snap Easy as... 1 2 3 [ java -jar newrelic_mysql_plugin.jar { "name" : "Localhost", "host" : "localhost", "user" : "USER_NAME_HERE", "passwd" : "USER_PASSWD_HERE" }, ] licenseKey=blahblahblah12345 Wednesday, November 6, 13
  32. 32. Independent runtime Infrastructure Varnish Server Memcached Server MySQL Server 1 MySQL Server 2 Externally monitored Platform Agents Varnish Agent Memcached Agent ... RPM MySQL Agent + GUID + New Relic Metric Store Wednesday, November 6, 13 MySQL UI ... Memcached UI Varnish UI
  33. 33. SaaS SaaS Service Customer A Customer B B Customer Customer B Data Customer B Data Data Customer X Customer X license key SaaS ‘agent code’ RPM + GUID + HTTP API New Relic Metric Store Wednesday, November 6, 13 SaaS UI ... Memcached UI Varnish UI
  34. 34. Don’t see one you like? Built 9 prior to launch, now at 14! Wednesday, November 6, 13 Built 6! Built 4! Built 3! Built the Redis Plugin in a couple hours, and used it to solve performance issues the same day
  35. 35. Simple APIs + SDKs = Easy to write Wednesday, November 6, 13
  36. 36. Simple APIs + SDKs = Easy to write •SDKs do all the “heavy lifting” •Java (primary), Ruby •Runs polling cycle •Connecting to New Relic •Aggregation Wednesday, November 6, 13
  37. 37. Simple APIs + SDKs = Easy to write •SDKs do all the “heavy lifting” •Java (primary), Ruby •Runs polling cycle •Connecting to New Relic •Aggregation •HTTP API •SaaS •Non Java/Ruby Wednesday, November 6, 13
  38. 38. Anatomy of a plugin agent (yes, it’s really that simple) public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }    private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }    public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();          reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    } } Wednesday, November 6, 13
  39. 39. Anatomy of a plugin agent (yes, it’s really that simple) public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    } Unique identifier (GUID), agent version    private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }    public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();          reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    } } Wednesday, November 6, 13
  40. 40. Anatomy of a plugin agent (yes, it’s really that simple) public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    } Unique identifier (GUID), agent version    private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    } Query metrics from system    public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();          reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    } } Wednesday, November 6, 13
  41. 41. Anatomy of a plugin agent (yes, it’s really that simple) public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    } Unique identifier (GUID), agent version    private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    } Query metrics from system    public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();   Value        reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    } } Wednesday, November 6, 13
  42. 42. Anatomy of a plugin agent (yes, it’s really that simple) public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    } Unique identifier (GUID), agent version    private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    } Query metrics from system    public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();   Identifier Value        reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    } } Wednesday, November 6, 13
  43. 43. Anatomy of a plugin agent (yes, it’s really that simple) public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    } Unique identifier (GUID), agent version    private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    } Query metrics from system    public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();   Identifier Units Value        reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    } } Wednesday, November 6, 13
  44. 44. Anatomy of a plugin agent (yes, it’s really that simple) public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    } Unique identifier (GUID), agent version    private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    } Query metrics from system    public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();   Send to NR Identifier Units Value        reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    } } Wednesday, November 6, 13
  45. 45. Easy to build UIs Wednesday, November 6, 13
  46. 46. Custom Dashboards Fast and easy UI creation Wednesday, November 6, 13
  47. 47. Optionally publish Lists in Plugin Central, makes available the UIs Wednesday, November 6, 13
  48. 48. Install UI = run the agent Infrastructure Varnish Server Memcached Server MySQL Server 1 MySQL Server 2 Externally monitored Platform Agents Varnish Agent Memcached Agent ... RPM MySQL Agent + GUID + New Relic Metric Store Wednesday, November 6, 13 MySQL UI ... Memcached UI Varnish UI
  49. 49. Install UI = run the agent Infrastructure Varnish Server Memcached Server MySQL Server 1 MySQL Server 2 Externally monitored Platform Agents Varnish Agent Memcached Agent ... RPM MySQL Agent + GUID + New Relic Metric Store Wednesday, November 6, 13 MySQL UI ... Memcached UI Varnish UI
  50. 50. “But Kevin, how much will all this cost?” Wednesday, November 6, 13
  51. 51. “But Kevin, how much will all this cost?” Free!! Wednesday, November 6, 13
  52. 52. New! Plugin community reviews Help improve plugins, give feedback Wednesday, November 6, 13
  53. 53. New! Private publishing Distribute within multiple accounts, closed Betas Wednesday, November 6, 13
  54. 54. In progress: Deployment recipes Simplified configuration and deployment across your data center Wednesday, November 6, 13
  55. 55. Get involved! ✓ Review plugins ✓ Contribute to the plugins • Many are open source • Write one! ✓ Contribute to the SDKs ✓ Help us with Chef • Try out our recipes • Try authoring recipes Wednesday, November 6, 13
  56. 56. Thank you! Wednesday, November 6, 13

×