Nick Galbreath http://client9.com/20130501 @ngalbreathCare and Feeding ofLarge ScaleGraphiteInstallationsNick Galbreath ★ ...
Nick Galbreath http://client9.com/20130501 @ngalbreathhttp://client9.com/20130501
Nick Galbreath http://client9.com/20130501 @ngalbreathWho is nickg?
Nick Galbreath http://client9.com/20130501 @ngalbreath(thats online advertising infrastructure)
Nick Galbreath http://client9.com/20130501 @ngalbreath• Over One Billion Points collected daily.• In "many" independent Gr...
so, graphite?
Nick Galbreath http://client9.com/20130501 @ngalbreathWho cares?• Making it easy to create, analyze andshare data can chan...
Nick Galbreath http://client9.com/20130501 @ngalbreathWhat is it you sayyou do here?• Your job is likely invisible to rest...
Nick Galbreath http://client9.com/20130501 @ngalbreathWhy Graphite?• Many innovations in each part of thestack• But, its t...
Nick Galbreath http://client9.com/20130501 @ngalbreathSharing in Caring• Allows data to be easily accessed• And easily sha...
What is it?
Nick Galbreath http://client9.com/20130501 @ngalbreathThe Big Picture• Its a database• But not ACID• And without all the d...
Nick Galbreath http://client9.com/20130501 @ngalbreathInstallation• 3 python-twistd servers• "carbon-cache"• "cache-aggreg...
Nick Galbreath http://client9.com/20130501 @ngalbreathBe Current• Dont use the OS default version.• Newer point releases o...
Nick Galbreath http://client9.com/20130501 @ngalbreathThe Documentation• everyone complains about it• historically bad, bu...
Nick Galbreath http://client9.com/20130501 @ngalbreathStorage Engine
Nick Galbreath http://client9.com/20130501 @ngalbreathWhat is it?• Storage engine• Handles reads, writes and creates of as...
Nick Galbreath http://client9.com/20130501 @ngalbreathGraphite Math• About 12 bytes per point.• Store 1 minute points for ...
Nick Galbreath http://client9.com/20130501 @ngalbreathDisk Layout• Each metric create a directory treeserver123.myapp.logi...
Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewarecarbon-cachePart 1
Nick Galbreath http://client9.com/20130501 @ngalbreathCarbon Cache• Metrics go in• csv file or python pickle format• TCP• M...
Nick Galbreath http://client9.com/20130501 @ngalbreathWrite Buffer• Most important feature is writebuffering to protect the ...
Nick Galbreath http://client9.com/20130501 @ngalbreathThe Cache• Its a write cache.• Once data is written,its out of the c...
Nick Galbreath http://client9.com/20130501 @ngalbreathNew Metrics• New metrics are created automatically• But, it is very ...
Nick Galbreath http://client9.com/20130501 @ngalbreathFALLOCATE• WHISPER_FALLOCATE_CREATE.=.True• Linux&Kernel&>=&2.6.23• ...
Nick Galbreath http://client9.com/20130501 @ngalbreathLimit the SizeLimit the size of the cache to avoidswapping or becomi...
Nick Galbreath http://client9.com/20130501 @ngalbreathGraphite for GraphiteBy&default,&carbon&itself&will&log&statistics&(...
Nick Galbreath http://client9.com/20130501 @ngalbreathStats onStats!
Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewarecarbon-aggregatorPart 2
Nick Galbreath http://client9.com/20130501 @ngalbreathPre-Aggregation• Sum or Average metrics based onwildcards and regexp...
Nick Galbreath http://client9.com/20130501 @ngalbreathdestination• r/w to localhost• split metrics to other aggregators• D...
Nick Galbreath http://client9.com/20130501 @ngalbreathAlong the way• renaming of metrics• whitelist and blacklist of aggre...
Nick Galbreath http://client9.com/20130501 @ngalbreathAlso...• has support for broadcasting data tomultiple downstream cac...
Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewarecarbon-relayPart 3
Nick Galbreath http://client9.com/20130501 @ngalbreathIts a Router!• Consistent Hashing (Sharding)• Or more rule-based rou...
Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewareStatsDPart 4
Nick Galbreath http://client9.com/20130501 @ngalbreathStatsD• https://github.com/etsy/statsd/• nodejs based but lots of ot...
Nick Galbreath http://client9.com/20130501 @ngalbreathStatsD use case• Its UDP based, so it excels atembedding a client in...
Nick Galbreath http://client9.com/20130501 @ngalbreathOf Note• https://github.com/armon/statsite• Need to look at this mor...
Nick Galbreath http://client9.com/20130501 @ngalbreathBackupsDo you really need them?http://bit.ly/11sPhNz
Nick Galbreath http://client9.com/20130501 @ngalbreathBackup• Doing naive backup causes graphiteperformance goes to crap.•...
Nick Galbreath http://client9.com/20130501 @ngalbreathDo you need to saveeverything?
Nick Galbreath http://client9.com/20130501 @ngalbreathnice isgoodhttp://www.beenthereyet.net/nice-franceIf you are doing y...
Nick Galbreath http://client9.com/20130501 @ngalbreathionice is betterIONICE(1) User Commands IONICE(1)NAMEionice - set or...
Nick Galbreath http://client9.com/20130501 @ngalbreathEven Better• Just write the metrics to two graphiteservers in your c...
Nick Galbreath http://client9.com/20130501 @ngalbreathMonitoring
Nick Galbreath http://client9.com/20130501 @ngalbreathWebUI• Hey, its a web server• do all the usual stuff• Ask for known s...
Nick Galbreath http://client9.com/20130501 @ngalbreathMistakes in URL or use offunctions cause Server 500
Nick Galbreath http://client9.com/20130501 @ngalbreathOld Stats• Dont forget to kill off old metrics• no updates in X days?...
Nick Galbreath http://client9.com/20130501 @ngalbreathMySQL• If you use SQLite3 -- uhh nothing tomonitor• If you use MySQL...
Nick Galbreath http://client9.com/20130501 @ngalbreathCPU is stable• ... except for apache usage• consider moving apache t...
Nick Galbreath http://client9.com/20130501 @ngalbreathDisk is SensitiveCompeting with otherprocesses for disk does this
Nick Galbreath http://client9.com/20130501 @ngalbreathMeans less written to diskMetrics updated
Nick Galbreath http://client9.com/20130501 @ngalbreathDangerous build upin cacheCache Size
Rendering
Nick Galbreath http://client9.com/20130501 @ngalbreathTune Apache• By default, your Apache install is likelyto be "unlimit...
Nick Galbreath http://client9.com/20130501 @ngalbreath/version/• Yes, ending "/" is required.• Ok not that exciting but ea...
Nick Galbreath http://client9.com/20130501 @ngalbreath/metrics/expand/• /metrics/expand/?query=server*• {"results": ["serv...
Nick Galbreath http://client9.com/20130501 @ngalbreath/events/• Ad-Hoc Events that dont deserve theirown metric type.• has...
Nick Galbreath http://client9.com/20130501 @ngalbreathThe WebUI• its "ok".. good for experiments• You will want to make yo...
Nick Galbreath http://client9.com/20130501 @ngalbreathWebUI Dashboards• The WebUI has a dashboard feature forloading and s...
Nick Galbreath http://client9.com/20130501 @ngalbreathGranularity• Like RRDTool, the resolution of thegraph depends on num...
Nick Galbreath http://client9.com/20130501 @ngalbreathVertical LineTechnology• Easy to make horizontal lines• Not so clear...
Nick Galbreath http://client9.com/20130501 @ngalbreathTurn "time since" intoeventsdrawAsInfinite(removeAboveValue(keepLast...
Nick Galbreath http://client9.com/20130501 @ngalbreath
Nick Galbreath http://client9.com/20130501 @ngalbreathTurnVersion Numbersinto EventsdrawAsInfinite(removeBelowValue(deriva...
Nick Galbreath http://client9.com/20130501 @ngalbreath
Nick Galbreath http://client9.com/20130501 @ngalbreathArbitrary LinesdrawAsInfinite(removeBelowValue(removeAboveValue(time...
Nick Galbreath http://client9.com/20130501 @ngalbreathReally Long URLs• Making graph but the URL is so longbrowsers are cl...
Nick Galbreath http://client9.com/20130501 @ngalbreathClient SideRendering• yeah...• works ok with a small number of point...
Nick Galbreath http://client9.com/20130501 @ngalbreathColors andChartJunk• Default color scheme is gross• Be kind to the h...
Nick Galbreath http://client9.com/20130501 @ngalbreathLooking forsomething to do?
Nick Galbreath http://client9.com/20130501 @ngalbreathAccelerate with PyPy• JIT for Python• ~ 5.9x performance improvement...
Nick Galbreath http://client9.com/20130501 @ngalbreathAccelerate with numpy• numpy provides fast vectormanipulation (C cod...
Nick Galbreath http://client9.com/20130501 @ngalbreathCeres Storage Engine• "Eventually Fixed Size" storage• More space effi...
Nick Galbreath http://client9.com/20130501 @ngalbreathOpenTSB• Not Graphite, but similar in spirit• Has "collectors" for b...
Nick Galbreath http://client9.com/20130501 @ngalbreathAdd More Functions• coursen (Im looking at you Ian Malpass,thats use...
Nick Galbreath http://client9.com/20130501 @ngalbreathMine the Apache Log• Which stats are used the most?• What are really...
Nick Galbreath http://client9.com/20130501 @ngalbreathAdd a TinyURLFeature• The URLs get really long and are hardto put in...
Nick Galbreath http://client9.com/20130501 @ngalbreathWrite Docsyeah you!
Nick Galbreath http://client9.com/20130501 @ngalbreathNick Galbreathhttp://www.client9.com/nickg@client9.comhttp://www.ipo...
Upcoming SlideShare
Loading in …5
×

Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013

1,943 views

Published on

Tip and tricks of using and scaling graphite. First presented at DevOpsDays Austin Texas 2013-05-01

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,943
On SlideShare
0
From Embeds
0
Number of Embeds
417
Actions
Shares
0
Downloads
28
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013

  1. 1. Nick Galbreath http://client9.com/20130501 @ngalbreathCare and Feeding ofLarge ScaleGraphiteInstallationsNick Galbreath ★ IPONWEBDevOpsDays ★ Austin Texas ★ 2013-04-30
  2. 2. Nick Galbreath http://client9.com/20130501 @ngalbreathhttp://client9.com/20130501
  3. 3. Nick Galbreath http://client9.com/20130501 @ngalbreathWho is nickg?
  4. 4. Nick Galbreath http://client9.com/20130501 @ngalbreath(thats online advertising infrastructure)
  5. 5. Nick Galbreath http://client9.com/20130501 @ngalbreath• Over One Billion Points collected daily.• In "many" independent Graphiteclusters.
  6. 6. so, graphite?
  7. 7. Nick Galbreath http://client9.com/20130501 @ngalbreathWho cares?• Making it easy to create, analyze andshare data can change yourorganization• Making a data-driven culture• Empowering developers, operations,qa, security and business to be moreconfident in the changes they make.
  8. 8. Nick Galbreath http://client9.com/20130501 @ngalbreathWhat is it you sayyou do here?• Your job is likely invisible to rest of theorganization• invisible things arent valued• so make what you do visible
  9. 9. Nick Galbreath http://client9.com/20130501 @ngalbreathWhy Graphite?• Many innovations in each part of thestack• But, its the Full Stack that really makesit special.• On-disk layout to UI to API to... thecommunity around it.
  10. 10. Nick Galbreath http://client9.com/20130501 @ngalbreathSharing in Caring• Allows data to be easily accessed• And easily shared. This makes itdifferent than many monitoringsolutions.• Its your own in-house mashupgenerator.
  11. 11. What is it?
  12. 12. Nick Galbreath http://client9.com/20130501 @ngalbreathThe Big Picture• Its a database• But not ACID• And without all the database tools
  13. 13. Nick Galbreath http://client9.com/20130501 @ngalbreathInstallation• 3 python-twistd servers• "carbon-cache"• "cache-aggregator"• "carbon-relay"• Apache / Django Web UI and API• Uses SQLite3/MySQL fordashboards / events
  14. 14. Nick Galbreath http://client9.com/20130501 @ngalbreathBe Current• Dont use the OS default version.• Newer point releases of graphite havesignificant improvements in storageengine and webui/api• Its 100% Python so"building it yourself" shouldnt to hard.• pip install works and is current.
  15. 15. Nick Galbreath http://client9.com/20130501 @ngalbreathThe Documentation• everyone complains about it• historically bad, but getting a lot better• Switched locations, but not allsearchengines are updated to use:http://graphite.readthedocs.org/• Source code is quite good, so RTFS
  16. 16. Nick Galbreath http://client9.com/20130501 @ngalbreathStorage Engine
  17. 17. Nick Galbreath http://client9.com/20130501 @ngalbreathWhat is it?• Storage engine• Handles reads, writes and creates of asingle metric to a fixed size file.• One file, kinda dumb (good).• Heres the API:https://github.com/graphite-project/whisper/blob/master/whisper.py
  18. 18. Nick Galbreath http://client9.com/20130501 @ngalbreathGraphite Math• About 12 bytes per point.• Store 1 minute points for 1 month and15 minutes for 11 months.• (60×24×30 + 4×24×30×11) ×12 =878kB• If you can keep all your points inmemory, then magic!
  19. 19. Nick Galbreath http://client9.com/20130501 @ngalbreathDisk Layout• Each metric create a directory treeserver123.myapp.logins.failed• Makes 3 directories• This creates a very branchy directorystructure• This has good and bad points.
  20. 20. Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewarecarbon-cachePart 1
  21. 21. Nick Galbreath http://client9.com/20130501 @ngalbreathCarbon Cache• Metrics go in• csv file or python pickle format• TCP• Metrics go to disk• whisper
  22. 22. Nick Galbreath http://client9.com/20130501 @ngalbreathWrite Buffer• Most important feature is writebuffering to protect the disk.• Data is buffered and written out onceper minute (or so).• But
  23. 23. Nick Galbreath http://client9.com/20130501 @ngalbreathThe Cache• Its a write cache.• Once data is written,its out of the cache• In other words, the cache ismetrics not on disk.• If the cache dies, you lose metrics• (btw: the read cache is the os diskcache)
  24. 24. Nick Galbreath http://client9.com/20130501 @ngalbreathNew Metrics• New metrics are created automatically• But, it is very expensive.• MAX_CREATES_PER_MINUTE.=.50• Saves your disk, but new metrics will"pile up" in cache.• May take 10m+ for your metrics to startflowing....
  25. 25. Nick Galbreath http://client9.com/20130501 @ngalbreathFALLOCATE• WHISPER_FALLOCATE_CREATE.=.True• Linux&Kernel&>=&2.6.23• fallocate is used to preallocate blocksto a file. For filesystems whichsupport the fallocate system call, thisis done quickly by allocating blocksand marking them as uninitialized,requiring no IO to the data blocks.This is much faster than creating afile by filling it with zeros.• https://bugs.launchpad.net/whisper/+bug/957827
  26. 26. Nick Galbreath http://client9.com/20130501 @ngalbreathLimit the SizeLimit the size of the cache to avoidswapping or becoming CPU bound.Sortsand serving cache queries gets moreexpensive as the cache grows.Use the value "inf" (infinity) for anunlimited cache size.MAX_CACHE_SIZE = infNo!.Infinity.does.not.exist.on.your.system!.&
  27. 27. Nick Galbreath http://client9.com/20130501 @ngalbreathGraphite for GraphiteBy&default,&carbon&itself&will&log&statistics&(such&as&a&count,metricsReceived)&with&the&top&level&prefix&of&carbon&at&an&interval&of&60seconds.&Set&CARBON_METRIC_INTERVAL&to&0&to&disable&instrumentationCARBON_METRIC_PREFIX&=&carbonCARBON_METRIC_INTERVAL&=&60
  28. 28. Nick Galbreath http://client9.com/20130501 @ngalbreathStats onStats!
  29. 29. Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewarecarbon-aggregatorPart 2
  30. 30. Nick Galbreath http://client9.com/20130501 @ngalbreathPre-Aggregation• Sum or Average metrics based onwildcards and regexps• Helps eliminate very slow queries onwebui• You can emit the final sum & all theindividual components or just the finalsum (via blacklists)
  31. 31. Nick Galbreath http://client9.com/20130501 @ngalbreathdestination• r/w to localhost• split metrics to other aggregators• Design your own system
  32. 32. Nick Galbreath http://client9.com/20130501 @ngalbreathAlong the way• renaming of metrics• whitelist and blacklist of aggregationand metrics
  33. 33. Nick Galbreath http://client9.com/20130501 @ngalbreathAlso...• has support for broadcasting data tomultiple downstream caches• but.. never used it.. and seems at oddswith the next middleware
  34. 34. Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewarecarbon-relayPart 3
  35. 35. Nick Galbreath http://client9.com/20130501 @ngalbreathIts a Router!• Consistent Hashing (Sharding)• Or more rule-based routing• Output to multiple carbon servershave not really used it much, but shouldwork similarly to scale outs ofmemcache, redis
  36. 36. Nick Galbreath http://client9.com/20130501 @ngalbreathMiddlewareStatsDPart 4
  37. 37. Nick Galbreath http://client9.com/20130501 @ngalbreathStatsD• https://github.com/etsy/statsd/• nodejs based but lots of otherimplementations• Receives UDP, send graphite-compatible output, flushed periodically.• Aggregation for all by default• Beside sum, also can compute otherbasic statistics (mean, 90% percentile),do sampling, have counters, etc.
  38. 38. Nick Galbreath http://client9.com/20130501 @ngalbreathStatsD use case• Its UDP based, so it excels atembedding a client inside theapplication• UDP cant block or break the sendingapplication• Not so good for bulk metrics• Use both! Can work together withaggregator.
  39. 39. Nick Galbreath http://client9.com/20130501 @ngalbreathOf Note• https://github.com/armon/statsite• Need to look at this more• c + libev based• modern time series algorithms• very flexible output
  40. 40. Nick Galbreath http://client9.com/20130501 @ngalbreathBackupsDo you really need them?http://bit.ly/11sPhNz
  41. 41. Nick Galbreath http://client9.com/20130501 @ngalbreathBackup• Doing naive backup causes graphiteperformance goes to crap.• File system cache is trashed• Metrics are not written to disk (lag)• If OOM occurs then you lose metrics.
  42. 42. Nick Galbreath http://client9.com/20130501 @ngalbreathDo you need to saveeverything?
  43. 43. Nick Galbreath http://client9.com/20130501 @ngalbreathnice isgoodhttp://www.beenthereyet.net/nice-franceIf you are doing your own backup....
  44. 44. Nick Galbreath http://client9.com/20130501 @ngalbreathionice is betterIONICE(1) User Commands IONICE(1)NAMEionice - set or get process I/O scheduling class and prioritySYNOPSISionice [-c class] [-n level] [-t] -p PID...ionice [-c class] [-n level] [-t] command [argument...]DESCRIPTIONThis program sets or gets the I/O scheduling class and priority for aprogram. If no arguments or just -p is given, ionice will query the current I/Oscheduling class and priority for that process.When command is given, ionice will run this command with the givenarguments. If no class is specified, then command will be executed with the"best-effort" scheduling class. The default priority level is 4.NOTESLinux supports I/O scheduling priorities and classes since 2.6.13 with theCFQ I/O scheduler.util-linux July 2011 IONICE(1)
  45. 45. Nick Galbreath http://client9.com/20130501 @ngalbreathEven Better• Just write the metrics to two graphiteservers in your client• Script to copy / resync "holes" whenrestoring.
  46. 46. Nick Galbreath http://client9.com/20130501 @ngalbreathMonitoring
  47. 47. Nick Galbreath http://client9.com/20130501 @ngalbreathWebUI• Hey, its a web server• do all the usual stuff• Ask for known stats,• check for 200• check for valid json output
  48. 48. Nick Galbreath http://client9.com/20130501 @ngalbreathMistakes in URL or use offunctions cause Server 500
  49. 49. Nick Galbreath http://client9.com/20130501 @ngalbreathOld Stats• Dont forget to kill off old metrics• no updates in X days? kill.• Exercise in "find" left to reader
  50. 50. Nick Galbreath http://client9.com/20130501 @ngalbreathMySQL• If you use SQLite3 -- uhh nothing tomonitor• If you use MySQL -- use the regularsuspects• And dont forget to backup!!
  51. 51. Nick Galbreath http://client9.com/20130501 @ngalbreathCPU is stable• ... except for apache usage• consider moving apache to separatemachine
  52. 52. Nick Galbreath http://client9.com/20130501 @ngalbreathDisk is SensitiveCompeting with otherprocesses for disk does this
  53. 53. Nick Galbreath http://client9.com/20130501 @ngalbreathMeans less written to diskMetrics updated
  54. 54. Nick Galbreath http://client9.com/20130501 @ngalbreathDangerous build upin cacheCache Size
  55. 55. Rendering
  56. 56. Nick Galbreath http://client9.com/20130501 @ngalbreathTune Apache• By default, your Apache install is likelyto be "unlimited" in CPU and Memoryusage.• Select a wildcard metric for a long timeperiod can easily turn a httpd processin 1GB. (this seems like a bug actually)• OOM death.
  57. 57. Nick Galbreath http://client9.com/20130501 @ngalbreath/version/• Yes, ending "/" is required.• Ok not that exciting but easy check
  58. 58. Nick Galbreath http://client9.com/20130501 @ngalbreath/metrics/expand/• /metrics/expand/?query=server*• {"results": ["server001", "server002", ... ]}
  59. 59. Nick Galbreath http://client9.com/20130501 @ngalbreath/events/• Ad-Hoc Events that dont deserve theirown metric type.• has tags, time, and text• Stored in SQLite3 by default by thewebapp.• Rest UI is primitive
  60. 60. Nick Galbreath http://client9.com/20130501 @ngalbreathThe WebUI• its "ok".. good for experiments• You will want to make your owndashboard.• Good news! The API is a URL, so itsvery easy.
  61. 61. Nick Galbreath http://client9.com/20130501 @ngalbreathWebUI Dashboards• The WebUI has a dashboard feature forloading and saving graphs• It saves data in SQLite3 by default• Since its there people will use it• So hack to remove it or, switch toMySQL.
  62. 62. Nick Galbreath http://client9.com/20130501 @ngalbreathGranularity• Like RRDTool, the resolution of thegraph depends on number of pixelsused. No sub-pixel rendering!• Rapid spikes can be "averaged away"in week-long views in small graphs.
  63. 63. Nick Galbreath http://client9.com/20130501 @ngalbreathVertical LineTechnology• Easy to make horizontal lines• Not so clear how to makead-hoc vertical lines
  64. 64. Nick Galbreath http://client9.com/20130501 @ngalbreathTurn "time since" intoeventsdrawAsInfinite(removeAboveValue(keepLastValue(YOURMETRIC),120))
  65. 65. Nick Galbreath http://client9.com/20130501 @ngalbreath
  66. 66. Nick Galbreath http://client9.com/20130501 @ngalbreathTurnVersion Numbersinto EventsdrawAsInfinite(removeBelowValue(derivative(keepLastValue(YOURMETRIC)),0.1))
  67. 67. Nick Galbreath http://client9.com/20130501 @ngalbreath
  68. 68. Nick Galbreath http://client9.com/20130501 @ngalbreathArbitrary LinesdrawAsInfinite(removeBelowValue(removeAboveValue(time("time"),timestamp),timestamp))
  69. 69. Nick Galbreath http://client9.com/20130501 @ngalbreathReally Long URLs• Making graph but the URL is so longbrowsers are clipping them?• Send query string data as a POST
  70. 70. Nick Galbreath http://client9.com/20130501 @ngalbreathClient SideRendering• yeah...• works ok with a small number of points• crashes existing browsers with largenumber of points• Server side faster in many cases!• Well try again in 2014
  71. 71. Nick Galbreath http://client9.com/20130501 @ngalbreathColors andChartJunk• Default color scheme is gross• Be kind to the handicapped (uhh, me)http://colorbrewer2.org/• Good overview here:http://bit.ly/10Hu7zU
  72. 72. Nick Galbreath http://client9.com/20130501 @ngalbreathLooking forsomething to do?
  73. 73. Nick Galbreath http://client9.com/20130501 @ngalbreathAccelerate with PyPy• JIT for Python• ~ 5.9x performance improvement• Actually works and is stable• Compatible with twisted and Django
  74. 74. Nick Galbreath http://client9.com/20130501 @ngalbreathAccelerate with numpy• numpy provides fast vectormanipulation (C code)• graphite web gui does a lot of vectormanipulation• hmmmm.....
  75. 75. Nick Galbreath http://client9.com/20130501 @ngalbreathCeres Storage Engine• "Eventually Fixed Size" storage• More space efficient == moreperformance• seehttp://blog.sweetiq.com/2013/01/using-ceres-as-the-back-end-database-to-graphite/
  76. 76. Nick Galbreath http://client9.com/20130501 @ngalbreathOpenTSB• Not Graphite, but similar in spirit• Has "collectors" for basic ops stats• Used by StumbleUpon, Box.net,pintrest• Good: Stores data in HBASE/Hadoop• Bad: Stores data in HBASE/Hadoop
  77. 77. Nick Galbreath http://client9.com/20130501 @ngalbreathAdd More Functions• coursen (Im looking at you Ian Malpass,thats useful for client-side rendering)• Real vertical lines (our hacks are stupid)• Better operators (would nice to knoweasily how many metrics you have, e.g.select count(*))
  78. 78. Nick Galbreath http://client9.com/20130501 @ngalbreathMine the Apache Log• Which stats are used the most?• What are really slow queries?• Can you optimize them?• What time frames are used?• How much old data do you really needto store?its in thequery string
  79. 79. Nick Galbreath http://client9.com/20130501 @ngalbreathAdd a TinyURLFeature• The URLs get really long and are hardto put into email, etc.• Make a tinyurl feature into the djangoapp and integrate into dashboard.
  80. 80. Nick Galbreath http://client9.com/20130501 @ngalbreathWrite Docsyeah you!
  81. 81. Nick Galbreath http://client9.com/20130501 @ngalbreathNick Galbreathhttp://www.client9.com/nickg@client9.comhttp://www.iponweb.com/ngalbreath@iponweb.netLets MakeSome Graphs!

×