Use the DTM pattern to allow developers to quickly add stats and easily searchable events to your application.

1. Logstash Pattern - DTM
1

2. Goal
1) Ensure that developers can quickly get comprehensive log and
performance data without operations intervention.
2) Avoid a complete rewrite of code.
3) Avoid brittle regexp parsing that may change often.
2

3. Solution – DTM
●
- DTM stands for nothing
●
- Just a short string that rarely appears in the languages that I support
●
- Good intermediate step
3

4. Fixes...
Log in JSON
- parsing is cheap
- format change is
fine
4

5. Example Usage
Calls to some 3rd party API are failing intermittently and/or taking too
long
How do I figure out what is going on when there are several hundred
requests a second and there are 40 app servers?
I don't have the time to re-write all of the logging code to use JSON
5

6. Answer...
Use DTM_EVENTS and DTM_STATS
6

7. Sample Code
7

8. Outputs the following log lines
2013-11-14T23:36:23.491037+00:00 app[web.1]:
{"DTM_EVENT":"Failure","fail":"BROKEN","user":"chris","group":"admins"}
2013-11-14T23:36:43.080817+00:00 app[web.1]:
{"DTM_STATS":"outside_api.fetch","value":0.089154052734,"stat_type":"timer"}
8

9. DTM_EVENTS
-JSON Blob
-Needs a unique DTM_EVENT field
{"DTM_EVENT":"Failure",
"fail":"BROKEN",
"user":"chris",
"group":"admins"}
9

10. Example Config (pre 1.3)
10

11. 11

12. DTM_STATS
- JSON blobs
Required fields
{"DTM_STATS":"outside_api.fetch",
1) DTM_STATS – stat name
"value":0.0001437664031982422,
"stat_type":"timer"}
2) value number
3) stat_type
timer, counter, gauge
12

13. DTM_STATS types
Counter Timer Timed events
Statsd provides the following
data count_ps count lower
mean_90 mean median std
sum_90 sum upper_90 upper
Rate events eg: number of
HTTP 200's a second
Gauge Number during a given time
period eg: 7 200's in the past
minute
13

14. Example Config (pre 1.3)
14

15. Can be found in graphite
stats.timers.dtm_stats.application.heroku.staging.outside_api.fetch.mean_90
stats.guage.dtm_stats.application.heroku.staging.STAT
stats.counter.dtm_stats.application.heroku.staging.STAT
15

16. Examples of Graphite Graphs
Can be saved under UserGraphs
16

17. More Information
Graphite on EC2
http://shokunin.co/blog/2013/03/16/graphite_on_ec2.html
Elasticsearch for Logstash
http://shokunin.co/blog/2013/06/24/elasticsearch.html
Map visualizations
http://shokunin.co/blog/2013/05/10/maptail_plus_logstash.html
17

18. Thanks!
Chris Mague
Shokunin.co
chris.mague@shokunin.co
IRC - #bosszaru
AIM - maguech
18