annalist - a statistics aggregator written in erlang

annalist,an Erlang stats aggregator Florian Odronitz @ Berlin Erlounge July 2012

Motivation• Application Statistics in Erlang• Count things (e.g. new users/day)• Record stats (e.g. 95 percentile of response time)• Pretty graphs• CSV for the Excel-folks

Concept• Store stuff in two granularities• Time and hierarchical tags• so count([<<“read”>>, <<“book”>>]) increments the counter in 14 keys

read/totalread/year/2012read/month/2012/07read/day/2012/07/24read/hour/2012/07/24/23read/minute/2012/07/24/23/01read/second/2012/07/24/23/01/22read/totalread/book/year/2012read/book/month/2012/07read/book/day/2012/07/24read/book/hour/2012/07/24/23read/book/minute/2012/07/24/23/01read/book/second/2012/07/24/23/01/22

Components• LevelDB stores data• eleveldb interfaces with LevelDB• uplevel provides bucket/key semantics and range queries• Cowboy for JSON API• sparks.js and Highstocks for graphs

LevelDB• Local, persistent key-value store• Ordered by key, iterations• Atomic writes and snapshots• Variable Cache Size

Features: Countingannalist:count([<<"create">>, <<"opinion">>])

Features: Counting

Features: Recordingannalist:record([<<"time">>, <<"get_user">>], 62.3)

Features: Recording

Features: Recording• Store distributions of values in a space efﬁcient manner with accuracy guaranties.• Targeted Quantiles:f_targeted([{0.05, 0.005}, {0.5, 0.02}, {0.95, 0.005}])Allow a maximum error of 0,5% for 5th and 95thpercentile and 2% error for median. Errors areerrors in rank.

Features: Recording

Features: Recording 1x107 1x106 100000 10000Value 1000 100 10 1 0 2000 4000 6000 8000 10000 12000 14000 Inserts

Features: Recording 80Compressed size (comp. every 10 inserts) 60 40 20 0 0 2000 4000 6000 8000 10000 12000 14000 Inserts

Features: API /annalist/hour_counts/ update%20user_location/ 2011/7/30/21/8640 ?callback=jQuery123jQuery123([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0...0,0,0,19,81,73,96,139,205,120,79,25,0,0,0,0,0,0,0,0,0,0,52,86,76,0,25,89,93,97,80,56,38,0,13,37,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,38,80,80,117,80,80,80,80,80,79,80,80,12,0,0,0,0,0,0,0,0,0,0,0,0,72,80,80,79,79,80,80,80,80,80,80,80,80,80,196,208,240,249,236,160,160,138,196,160,159,162,93,80,78,81,79,80,80,80,81,49,81,81,62,3,41,84,82,100,134,80,79,14,80,80,80,80,80,80,80,80,79,80,80,67,60,68,158,79,77,211,242,231,209,80,105,161,206,240,240,238,240,237,239,239,239,239,241,148,80,80,80,80,80,80,80,80,134,108,80,90,205,222,85,79,80,80,79,81,77,80,81,79,82,84,79,175,293,239,116,80,78,79,186,232,126,142,22,163,177,175,176,164,168,167,166,180,177,173,168,34,0,0,109,179,180,176,178,162,180,174,173,163,175,166,173,173,178,179,164,130,0,0,0,0,21,11,0,0,0,17,44,141,145,162,73,0,0,0,0,5,118,0,6,0,197,232,0,69,80,80,80,78,28,24,0,49,80,72,75,70,2,0,0,0,0,0,0,0,24,166,373,290,309,260,275,232,123,138,132,80,79,95,79,125,132,80,79,80,79,81,80,79,113,99,101,92,139,204,244,250,224,194,166,80,223,188,80,80,78,73,80,79,35,0,0,9,148,169,301,300,304,263,109,251,272,171,138,114,183,184,12,0,0,0,0,0,0,0,0,0,189,299,145,114,184,174,196,225,241,264,231,52,85,163,166,50,3,11,15,9,29,55,19,84,149,132,115,90,68,56,97,112,235,315,301,289,274,223,242,273,260,260,293,308,294,189,180,187,242,198,178,225,248,248,244,433,359,227,180,223,206,180,179,180,180,180,179,181,179,184,241,131,227,322,310, 286,260,271,327,348,319,187,161,108,79,80,0,0,0,0,0,0,0,44,166,149,148,13,0,41,81,25,0,166,49,4,113,107,0]);

Characteristic

Characteristic• Sparse counting• No backpressure• 1 gen_server, 1 CPU• eventually consistent

Performance• Counting: ~ 4k ops/sec with ([<<“a”>>, <<“b”>>]) equals ~ 56k increments/sec• Stats: ~ 690 ops/sec with ([<<“a”>>, <<“b”>>], 1) equals ~ 9.66k recordings/sec

Future• HTTP API for stats• CSV Export• Purging• Variable Timeslots• Multiple Workers (CAS with locks in ets?)

Thanks• leveldb.googlecode.com• https://github.com/basho/eleveldb• https://github.com/odo/uplevel• https://github.com/odo/annalist• https://github.com/extend/cowboy/• http://omnipotent.net/jquery.sparkline/• highcharts.com

annalist - a statistics aggregator written in erlang

by odo101 on Jul 26, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

annalist - a statistics aggregator written in erlang Presentation Transcript