Making Pretty Charts in Splunk


Published on

Splunk Users' Conference 2010 session: how to effectively use Splunk to create compelling charts

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Welcome!
  • Intro
  • There are 5 main steps you generally go through before you can have a meaningful and pleasing chart. Today we’re going to cover all 5 steps in Splunk. But before that, can you answer the obvious question…
  • Do you actually have a reason for making a chart? What is the desired outcome? -- want some business intelligence? -- need to monitor something? -- or just for the hell of it? If you don’t know…
  • Might I suggest -- can make lots of different charts -- can share them with friends -- can use pretty colors At least have a purpose in mind.
  • Step 1: do you prep work. -- due-diligence -- mis-en-place The right questions can make your charting easier.
  • Question #1: what data do you have at your disposal? -- Splunk can index anything -- any text, stream, file, packet, script Who owns the data? Are you going to have to ask permission?
  • Question #2: Who is going to read it? -- Consider your audience when picking data to display. -- Technical staff generally want detailed information. -- Your boss may not. What is the reader going to do with it? -- make business decisions -- incidence response -- feel good -- look at pretty pictures -- be able to assign blame There was a story about a potential customer who was so excited about Splunk, threw everything at it. Then tried to discover who was to blame for a current production problem. Found out it was his friend. Radio silence. Splunk was that good.
  • Think about what what format you want. More importantly, what will the caption be? You may end up a a slave to a less than desirable format if you just wing it. This chart: General Stanley McChrystal declared that "When we understand that slide, we'll have won the war" at a briefing in Kabul last summer Does not reflect well upon you.
  • One way to visualize is to think about the big picture first.
  • This is an example VP-level dashboard built entirely in Splunk * 60-day period * Shows medium-term trending; influences business decisions * dashboard.html template * custom view-specific CSS * HiddenSavedSearch module * HiddenChartFormatter module * SingleValue module
  • This is a real-time dashboard intended for a NOC situation -- everything is on a 30-second window -- shows info that people may need to react to quickly I’ll cover both of those examples later in the talk.
  • Note that both previous examples used tables! -- “visualization” doesn’t really just mean charts -- tables can be far more efficient at displaying certain kinds of data than charts
  • Even simpler are single numeric values -- Apple’s Q2 revenue for example -- if context is well known, there’s no need to complicate matters with “chartjunk”
  • OK, so on to charts. These are all native chart types that Splunk can render. I’ll be brief; I want to highlight how you match chart types to data. The column chart: -- great for inspecting discrete data -- can easily compare single value series on common axis -- identify trends
  • The line chart: -- great for comparing multiple series -- compare on similar Y-axis
  • Split line chart: -- variant of the line series -- great for when trending of individual series is wildly different
  • The area chart -- use to track multiple series in relationship to each other -- in 100% form, very quickly see proportional changes over time
  • The pie chart: do not use. -- cannot compare 2 slices together easily -- no common point of reference -- cannot determine distribution
  • A column chart lets you actually see the data on comparable terms -- ‘Georgian’ is more than half the value of ‘Hawaiian’ -- "The only worse design than a pie chart is several of them.“ – Edward Tufte You can see the trend among data (in this case it’s exponential). Why is trending important?
  • Example: -- in basic medical triage you must always record vitals over time -- walking into ER and only saying your pulse is 90 is useless -- you need to know if it’s rising or falling, or stable So what are you trending over? Well Splunk likes time…
  • We’re going to focus on time-based ranges and the main modes of trending: Real-time: -- still one of the coolest things about splunk -- even if you know nothing about the data, it’s still cool to see stuff come in real-time What is it good for? -- things that require immediate response -- when you only need to see at most an hour of content; 30 seconds is also useful Examples: -- network operations -- security operations -- just in time operations
  • The flip side is historical; what people typically expect What is it good for? -- making business decisions based on data -- 7, 30 day moving trends Examples: -- infrastructure planning -- bandwidth usage -- peak/off-peak tracking
  • Finally! We cover the tools in Splunk that can make effective charts.
  • How many people are familiar with the report builder? The standard report builder is accessible from the search interface -- you can start charting searches that are still in flight -- easy dropdown-based chart building -- handles simple cases
  • How many people are familiar with the advanced charting view? Advanced charting is where most of us like to chart -- direct access to the search language -- tabular view below -- has common set of charting controls
  • The search language is inspired by the UNIX command line -- typically the first command is assumed to be the ‘search’ command -- any number of commands can be chained together -- there are over 100 search commands that come with Splunk How many people are familiar with the search language? Novice? Intermediate? Advanced? There were a bunch of sessions on the search language (check your preso material if you didn’t attend) I’ll go through some of the workhorse commands used to generate charts…
  • Splunk search results are nothing more than a big table of data -- the event text is copied into the ‘_raw’ field -- it’s just another field in the result set -- if you understand this, you will grasp the search language easily -- the UI depends on underscore fields (which are not displayed) Knowing that this is just a table, you can use search language to transform the results any which way you want
  • This is the heavy hitter for any IT ops commands -- shows you over time what something is doing -- can take any of the stats commands and generate multiple series -- can control the granularity of the bucketing
  • Here is an example of the timechart command that looks at Splunk internal components over time -- uses automatic defaults to determine sensible time buckets
  • The ‘top’ command does what it sounds like -- displays the top values of any field in your results -- can do top n combinations -- can specify how many ordered items to return
  • Display is great when paired with a bar chart, or column -- please don’t use pie charts (will cover that in a bit)
  • This is the generic version of timechart -- behind the scenes, this powers ‘timechart’ -- in essence, plots some function of field A by field B -- like ‘timechart’, you can actually invoke the eval() command
  • This example shows the average bytes transferred over a time period split by client IP addresses -- this is over every piece of data the Splunk knows about -- you can restrict by time window by just setting the time range
  • The contingency table is not a graphical command, but is equally as powerful -- contingency is used very often in statistical analysis: determine if variable X really has an effect on measured property Y -- essentially is a counter: will tally combinations of X/Y
  • Note that the Y is ‘clientip’ and X is ‘http_status’; the numbers in the middle of the table simply show the number of occurences -- you can then apply the heatmap decorator to visually differentiate hotspots
  • There are lots of other commands than can manipulate the data any which way you want.
  • Let Splunk do your dirty work -- automation can alleviate much of the manual labor -- no reason to always check splunk when it can email you
  • Scheduling a saved search is the best practice -- defer work to off-peak hours -- allow multiple users to share results -- have Splunk alert you when certain conditions are met -- receive results via email, RSS plug: customizing and using scripted alerts Plug: monitoring with splunk
  • Assembling multiple saved searches on a dashboard is great for overviews -- use simple dashboard creator -- takes existing saved searches and lets you arrange onto a dashboard -- don’t cram a ton of searches onto a dashboard: use multiple dashboards!
  • The main areas of presentation-layer customizations fall into: -- view-level CSS: full access to override any CSS rules -- chart-level properties: each chart can have its own formatting
  • This is the before picture: -- default CSS -- default charting properties -- uses the standard panels -- uses standard singleValue modules -- uses default charting properties: note the axis labels, legend, colors, etc
  • You can have as many different CSS files on a per-app basis -- each view can reference any CSS file that is available in its app -- not going to cover CSS customization here; plenty of online resources -- recommend Firebug+Firefox or Chrome+Web Inspector
  • After: -- this is simple CSS and charting properties applied -- no other structural changes involved Source is available online
  • There are many different properties you can adjust on a per-chart basis -- these are set in XML configuration -- each property name is hierarchical -- dots are used to denote hierarchy -- --
  • * Add link to advanced XML
  • Promote usability sessions! We’re hiring
  • Making Pretty Charts in Splunk

    1. 1. Building pretty charts
    2. 2. Johnvey Hwang Splunk Inc.
    3. 3. Agenda <ul><li>Prep </li></ul><ul><li>Visualize </li></ul><ul><li>Build it </li></ul><ul><li>Automate </li></ul><ul><li>Polish </li></ul>
    4. 4. Q: Why are you making a chart? COMM153: Business Graphics, Fall 2010, Midterm exam
    5. 5.
    6. 6. Step 1: Prep
    7. 7. What data is available?
    8. 8. Who is going to read it?
    9. 10. Decision Step 2: Visualize
    10. 12. Finished NOC dashboard
    11. 13. Tables are sexy.
    12. 14. Q2 Revenue $17.5B So are numbers.
    13. 15. Column chart
    14. 16. Line chart
    15. 17. Line chart (split)
    16. 18. Area chart (100% stacked)
    17. 19. Is ‘ Georgian ’ half or a third of ‘ Hawaiian ’? Is this linear, geometric, or exponential? Pie charts suck
    18. 20. Pie chart remedy
    19. 21. Trending is better Pulse = 90 vs.  
    20. 22. Time range: real-time Requires immediate response 30 second – 1 hour window
    21. 23. Time range: historical Multiple day – month window Time to respond is in weeks or months
    22. 24. Step 3: Build it
    23. 25. Report builder
    24. 26. Advanced charting view See search box and chart on one page
    25. 27. Search language: overview index=main error datacenter=SFO | timechart avg(bytes) by host event search clause transforming clause Over 100 search commands…
    26. 28. The dataset _time _raw host field1 18283495832 2010-08-10 08:52:01 ERROR: something went wrong on server Prod_apache_1 ERROR 18383827123 2010-08-10 08:52:01 INFO: redirect to a better page Prod_apache_2 INFO
    27. 29. Search language: timechart … | timechart span=1h count “ count the number of events per hour” … | timechart avg(delay) by host “ calculate the average delay and track each host separately” … | timechart avg(delay) min(delay) max(delay) “ calculate the average, minimum, and maximum delays per auto-bucket”
    28. 30. Search language: timechart index=_internal | timechart count by group
    29. 31. Search language: top … | top limit=50 users “ list the top 50 users” … | top major_version, minor_version “ list the top (10) combinations of major and minor versions” … | top source by host “ list the top sources grouped by most frequent host”
    30. 32. Search language: top index=_internal source=&quot;*access.log&quot; | top uri_path | fields uri_path count
    31. 33. Search language: chart … | chart avg(delay) by sender “ list the average delay for every sender” … | chart max(bytes) over clientip by uri useother=f “ list the maximum bytes of the top ‘uri’ for every ‘clientip’”
    32. 34. Search language: chart index=_internal source=&quot;*access.log&quot; | chart avg(bytes) by clientip
    33. 35. Search language: ctable … | ctable clientip http_status “ list every combination of ‘clientip’ and ‘http_status’ and their freqencies” … | ctable clientip http_status maxcols=10 “… restrict to a max of 10 http_status columns’”
    34. 36. Search language: ctable index=_internal source=&quot;*access.log&quot; | ctable clientip status maxcols=10 maxrows=10
    35. 37. Search language: more stats : average, min, max, stdev, distinct count, mode, variance,… streamstats : calculate running statistics up to current event rangemap : bucket results into ranges like ‘low’, ‘medium’, ‘high’ kmeans : partition results into k-means clusters trendline : calculate moving averages accum : creates a new field of running total of any field
    36. 38. Step 4: Automate
    37. 39. Schedule it <ul><li>Let Splunk run the search on your behalf </li></ul><ul><li>View results immediately </li></ul><ul><li>Avoid unnecessary server load </li></ul><ul><li>Receive automated alerts </li></ul>
    38. 40. Assemble dashboards
    39. 41. Step 5: Polish
    40. 42. What can you change? View CSS Chart properties
    41. 44. Custom CSS 1. Add custom CSS files to the app: $SPLUNK_HOME/etc/apps/<APP_NAME>/appserver/static/<FOO>.css 2. Restart splunkweb (only on first create) 3. Add to view: <view stylesheet=“FOO.css”> 4. Save to: $SPLUNK_HOME/etc/apps/<APP_NAME>/default/data/ui/views/<NAME>.css
    42. 46. Charting properties Hundreds of different properties available Ex: Common property to change tick label visibility: charting.primaryAxis.majorLabelVisibility = hide Ex: Area chart type-specific series stacking mode: charting.AreaChart.stackMode = stacked100 Ex: Common legend label width: charting.legend.labelStyle.maximumWidth = 500
    43. 48. Simple XML form <dashboard> <label>My dashboard</label> <row> <chart> <searchName>My saved report</searchName> <option name=&quot; charting.seriesColors &quot;> [0xFF0000,0xFFFF00,0x00FF00] </option> <option name=&quot; charting.legend.placement &quot;> top </option> </chart> </row> </dashboard>
    44. 49. Advanced XML form ... <module name=&quot;HiddenChartFormatter&quot;> <param name=&quot; charting.seriesColors &quot;> [0xFF0000,0xFFFF00,0x00FF00] </param> <param name=&quot; charting.legend.placement &quot;> top </param> <module name=&quot;FlashChart&quot; /> </module> ...
    45. 50. More help Charting reference documentation Splunk community Q&A site Edward Tufte - “Father of data visualization” Blogs for inspiration http:// Demo material