annalist,an Erlang stats aggregator  Florian Odronitz @ Berlin Erlounge July 2012
Motivation• Application Statistics in Erlang• Count things (e.g. new users/day)• Record stats (e.g. 95 percentile of respo...
Concept• Store stuff in two granularities• Time and hierarchical tags• so count([<<“read”>>, <<“book”>>]) increments the c...
read/totalread/year/2012read/month/2012/07read/day/2012/07/24read/hour/2012/07/24/23read/minute/2012/07/24/23/01read/secon...
Components• LevelDB stores data• eleveldb interfaces with LevelDB• uplevel provides bucket/key semantics and  range querie...
LevelDB• Local, persistent key-value store• Ordered by key, iterations• Atomic writes and snapshots• Variable Cache Size
Features: Countingannalist:count([<<"create">>, <<"opinion">>])
Features: Counting
Features: Counting
Features: Recordingannalist:record([<<"time">>, <<"get_user">>], 62.3)
Features: Recording
Features: Recording• Store distributions of values in a space efficient  manner with accuracy guaranties.• Targeted Quantil...
Features: Recording
Features: Recording         1x107         1x106        100000         10000Value          1000           100            10...
Features: Recording         1x107         1x106        100000         10000Value          1000           100            10...
Features: Recording                                           80Compressed size (comp. every 10 inserts)                  ...
Features: API                              /annalist/hour_counts/                              update%20user_location/    ...
Characteristic
Characteristic• Sparse counting• No backpressure• 1 gen_server, 1 CPU• eventually consistent
Performance• Counting:  ~ 4k ops/sec with ([<<“a”>>, <<“b”>>])  equals ~ 56k increments/sec• Stats:  ~ 690 ops/sec with ([...
Future• HTTP API for stats• CSV Export• Purging• Variable Timeslots• Multiple Workers (CAS with locks in ets?)
Thanks• leveldb.googlecode.com• https://github.com/basho/eleveldb• https://github.com/odo/uplevel• https://github.com/odo/...
Upcoming SlideShare
Loading in...5
×

annalist - a statistics aggregator written in erlang

756

Published on

Published in: Technology, News & Politics
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
756
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • annalist - a statistics aggregator written in erlang

    1. 1. annalist,an Erlang stats aggregator Florian Odronitz @ Berlin Erlounge July 2012
    2. 2. Motivation• Application Statistics in Erlang• Count things (e.g. new users/day)• Record stats (e.g. 95 percentile of response time)• Pretty graphs• CSV for the Excel-folks
    3. 3. Concept• Store stuff in two granularities• Time and hierarchical tags• so count([<<“read”>>, <<“book”>>]) increments the counter in 14 keys
    4. 4. read/totalread/year/2012read/month/2012/07read/day/2012/07/24read/hour/2012/07/24/23read/minute/2012/07/24/23/01read/second/2012/07/24/23/01/22read/totalread/book/year/2012read/book/month/2012/07read/book/day/2012/07/24read/book/hour/2012/07/24/23read/book/minute/2012/07/24/23/01read/book/second/2012/07/24/23/01/22
    5. 5. Components• LevelDB stores data• eleveldb interfaces with LevelDB• uplevel provides bucket/key semantics and range queries• Cowboy for JSON API• sparks.js and Highstocks for graphs
    6. 6. LevelDB• Local, persistent key-value store• Ordered by key, iterations• Atomic writes and snapshots• Variable Cache Size
    7. 7. Features: Countingannalist:count([<<"create">>, <<"opinion">>])
    8. 8. Features: Counting
    9. 9. Features: Counting
    10. 10. Features: Recordingannalist:record([<<"time">>, <<"get_user">>], 62.3)
    11. 11. Features: Recording
    12. 12. Features: Recording• Store distributions of values in a space efficient manner with accuracy guaranties.• Targeted Quantiles:f_targeted([{0.05, 0.005}, {0.5, 0.02}, {0.95, 0.005}])Allow a maximum error of 0,5% for 5th and 95thpercentile and 2% error for median. Errors areerrors in rank.
    13. 13. Features: Recording
    14. 14. Features: Recording 1x107 1x106 100000 10000Value 1000 100 10 1 0 2000 4000 6000 8000 10000 12000 14000 Inserts
    15. 15. Features: Recording 1x107 1x106 100000 10000Value 1000 100 10 1 0 2000 4000 6000 8000 10000 12000 14000 Inserts
    16. 16. Features: Recording 80Compressed size (comp. every 10 inserts) 60 40 20 0 0 2000 4000 6000 8000 10000 12000 14000 Inserts
    17. 17. Features: API /annalist/hour_counts/ update%20user_location/ 2011/7/30/21/8640 ?callback=jQuery123jQuery123([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0...0,0,0,19,81,73,96,139,205,120,79,25,0,0,0,0,0,0,0,0,0,0,52,86,76,0,25,89,93,97,80,56,38,0,13,37,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,38,80,80,117,80,80,80,80,80,79,80,80,12,0,0,0,0,0,0,0,0,0,0,0,0,72,80,80,79,79,80,80,80,80,80,80,80,80,80,196,208,240,249,236,160,160,138,196,160,159,162,93,80,78,81,79,80,80,80,81,49,81,81,62,3,41,84,82,100,134,80,79,14,80,80,80,80,80,80,80,80,79,80,80,67,60,68,158,79,77,211,242,231,209,80,105,161,206,240,240,238,240,237,239,239,239,239,241,148,80,80,80,80,80,80,80,80,134,108,80,90,205,222,85,79,80,80,79,81,77,80,81,79,82,84,79,175,293,239,116,80,78,79,186,232,126,142,22,163,177,175,176,164,168,167,166,180,177,173,168,34,0,0,109,179,180,176,178,162,180,174,173,163,175,166,173,173,178,179,164,130,0,0,0,0,21,11,0,0,0,17,44,141,145,162,73,0,0,0,0,5,118,0,6,0,197,232,0,69,80,80,80,78,28,24,0,49,80,72,75,70,2,0,0,0,0,0,0,0,24,166,373,290,309,260,275,232,123,138,132,80,79,95,79,125,132,80,79,80,79,81,80,79,113,99,101,92,139,204,244,250,224,194,166,80,223,188,80,80,78,73,80,79,35,0,0,9,148,169,301,300,304,263,109,251,272,171,138,114,183,184,12,0,0,0,0,0,0,0,0,0,189,299,145,114,184,174,196,225,241,264,231,52,85,163,166,50,3,11,15,9,29,55,19,84,149,132,115,90,68,56,97,112,235,315,301,289,274,223,242,273,260,260,293,308,294,189,180,187,242,198,178,225,248,248,244,433,359,227,180,223,206,180,179,180,180,180,179,181,179,184,241,131,227,322,310, 286,260,271,327,348,319,187,161,108,79,80,0,0,0,0,0,0,0,44,166,149,148,13,0,41,81,25,0,166,49,4,113,107,0]);
    18. 18. Characteristic
    19. 19. Characteristic• Sparse counting• No backpressure• 1 gen_server, 1 CPU• eventually consistent
    20. 20. Performance• Counting: ~ 4k ops/sec with ([<<“a”>>, <<“b”>>]) equals ~ 56k increments/sec• Stats: ~ 690 ops/sec with ([<<“a”>>, <<“b”>>], 1) equals ~ 9.66k recordings/sec
    21. 21. Future• HTTP API for stats• CSV Export• Purging• Variable Timeslots• Multiple Workers (CAS with locks in ets?)
    22. 22. Thanks• leveldb.googlecode.com• https://github.com/basho/eleveldb• https://github.com/odo/uplevel• https://github.com/odo/annalist• https://github.com/extend/cowboy/• http://omnipotent.net/jquery.sparkline/• highcharts.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×