Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Recorded Future Blog Posts by Twain Taylor


Published on

Content sources:,,

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Recorded Future Blog Posts by Twain Taylor

  1. 1. The Role of Visualization in Cyber Intelligence The Recorded Future Blog This is the first post in our blog series, “Visualizing Cyber Intelligence.” The weapons have changed, the battlefield is different, but the war rages on. As technology has advanced, the war between cyber security professionals and cyber criminals has changed shape every few years. The criminals are becoming more sophisticated, forcing security professionals to continually step up their game. In the early days, computer worms would attack thousands of systems, and make it to the news headlines simply because of the scale of the attack. They are now passe. Laser-focussed attacks on companies’ confidential data are now the norm. The recent Target data breach is one example where the personal information of about 70 million customers was compromised. In fact, no enterprise is completely secure. Every enterprise has dealt with a data breach at some time. Interestingly, it’s not just the enterprises. Symantec reports 31% of attacks are on companies with less than 250 employees. Businesses today can’t afford to compromise on cyber intelligence to equip their security professionals. Security professionals have a hard time foreseeing new attacks because of how unique each one is. The amount of data to be analyzed has drastically increased. Not only the volume, but the variety of sources, and data types has made analysis difficult. Big data technologies are making it easier to capture and store data. However, analysis is still a challenge. Security Visualization is the Dark Horse 1/5
  2. 2. Raffael Marty, author of Applied Security Visualization, sees data visualization as the solution to this problem. In his book, he begins by talking about the problems plaguing security visualization today. These visualizations are either the work of designers with no background in security, or of security professionals who don’t understand data visualization. One is beautiful but not effective in getting work done, and the other is effective, but rather clunky. But what’s it about visualizations that makes them ideal to solve the big problem of security? Let’s look at the example of Anscombe’s quartet to understand this. Below are four data series that can be analyzed by any statistics tool. Now, these data sets have the same mean, variance, regression lines, and error rates. However, plotting them as charts makes their unique patterns become obvious. 2/5
  3. 3. You can effortlessly notice the different patterns in each data set. This simple example shows the power of a visual when analyzing data. Security visualization, or SecViz as Marty calls it, lies at the intersection of four major fields of study: security, statistics, computer science, and data visualization. Security professionals are well-versed in the first three disciplines. However, their level of competence in data visualization can be surprisingly low. Take, for example, the security visualization examples below. 3/5
  4. 4. Left: This happens when trying to use every shiny new feature of the visualization tool, and ignoring the purpose of the visual. Sometimes flashy looks can be such a priority, that it gets in the way of good design. Right: As is often the case, this dashboard seems like it’s been bolted on to a security tool as a trivial afterthought. These examples break the elementary rules of data visualization (which we’ll be covering in this series), and as a 4/5
  5. 5. result, hamper the work of a security professional. These approaches should be avoided, and replaced with a sound understanding of how data visualization works. That’s the goal of this series, “Visualizing Cyber Intelligence.” It will equip you with vital skills in data visualization that you can use daily as you fight cyber crime. Edward Tufte’s Concept of “Chartjunk” Let’s begin by looking at one of the core principles behind the work of Edward Tufte, author of a classic data visualization book, The Visual Display of Quantitative Information. In it, he defines a great data visualization as one that conveys the “most ideas, with least ink, in least space, least time.” Many charts use decor, and interactive features that distract the viewer from the actual data. This approach was frowned upon by Tufte, who termed these distractions as “chartjunk.” He devotes a major part of his book to fighting chartjunk. Tufte suggests an effective way of avoiding chartjunk is to reduce the amount of “non-data ink.” Data ink would include those parts of a chart that represent the data. Non-data ink would be the elements of the chart like textures and patterns, gridlines in the background, 3D enhancements, garish font styles, and the like. He gives the following example of a chart loaded with chartjunk, and then a better representation for the same chart without the chartjunk. They both plot the same data, but the second chart is a lot easier to analyze. All of us come across charts like this regularly. Going forward, be sure to look out for charts with excessive chartjunk. Consider how you can prune the charts you use on a daily basis to remove chartjunk, and highlight the data it represents. Tufte’s Sparkline Charts Tufte is considered a pioneer in the field of data visualization. Perhaps his most significant contribution to the field has been Sparkline charts. Sparklines are tiny, word-sized charts that can be embeded within a paragraph of text. They are a great example of maximizing data ink, and minimizing chartjunk. They’re widely used in stock market dashboards to plot the trend of many stock tickers in limited screen space. Think about how you can use sparklines to replace bigger charts in your dashboards. They can save you time by reducing clicks within a dashboard, and give you more information quicker. To conclude, this principle of maximizing data ink, and minimizing non-data ink, can save you from making a lot of elementary mistakes with data visualization. Stick around for the next post in this series. We’ll be discussing Ben Shneiderman’s “information-seeking mantra.” Author Profile: Twain Taylor is a guest blogger for Recorded Future. You can find more insight by Twain about the intersection between data visualization and big data on the FusionCharts blog. 5/5
  6. 6. How to Use the Information-Seeking Mantra in Cyber Intelligence Dashboards The Recorded Future Blog This is the second post in our blog series, “Visualizing Cyber Intelligence.” In the previous post, we got a glimpse of two important contributions of Edward Tufte to the field of data visualization: chartjunk and sparkline charts. Today, we’ll be looking at another data visualization guru whose work can have a profound impact on your cyber intelligence project. We’ll be discussing Ben Shneiderman’s information- seeking mantra. The Problem of Information Overload Today’s cyber security professional is dealing with volumes of data unlike ever before. Just about every analyst deals with information overload on a daily basis. Data visualization has always been used to make sense of huge quantities of data, and draw patterns that don’t stand out when viewing the raw data set. However, many dashboards still use text-based interfaces that overwhelm its viewers, and prevent him or her from taking action. In today’s big data world, it’s imperative to employ proper data visualization techniques. It’s a crime not to. To get started, let’s look at an example of a dashboard that suffers from information overload. 1/7
  7. 7. This dashboard, which is actually a table in disguise, overloads its viewer with too much information right up front. It screams, “I don’t care what you want, this is what you’ll get!” It gets straight to the minute details without informing the viewer of the broader trends in the data. It doesn’t make a point. It doesn’t allow the viewer to get an overview first, and then decide which area of the dashboard they’d like to further investigate. And this is with just 10 rows of data. If this data were to run into the thousands, or hundreds of thousands, which is common in cyber intelligence scenarios, this kind of dashboard would be a nightmare to use. Yet, how often do cyber intelligence analysts make do with tables, and poorly designed dashboards, that hide the story behind the data? The Information-Seeking Mantra In dealing with the problem of information overload, Ben Shneiderman’s work has been a landmark in the recent evolution of data visualization. Through his research, Shneiderman noticed the most powerful visualizations share a common trait, or mantra: overview first, zoom and filter, then details-on-demand. This is Shneiderman’s “information- seeking mantra.” Let’s discuss the mantra in detail, and see how to apply it when creating a dashboard. Overview First The most important part of a dashboard is the “overview” section. It’s the first thing a viewer sees in the dashboard, and guides the him or her to other parts of the product for further exploration. When designing a dashboard, maximum time should be spent on perfecting, and fine-tuning the overview section. The overview should summarize the overarching story from the entire data set without getting into the minor details. It shouldn’t overload the user with too much data, which is where interactive charts, gauges, and maps serve to 2/7
  8. 8. reduce data clutter, and bring out the story more powerfully. At the same time it shouldn’t leave out important parts of the story by using just a single pie chart, and hiding all the data a layer deeper. Often, great dashboards use a combination of chart types like the line chart, bar chart, maps, and gauges to give the viewer variety, and clarity when studying the data. The overview section should be carefully planned to highlight the important parts of the story, and give lesser weight to the not-so-critical parts. To do this, you may want to organize the entire section into many sub-sections that are clearly labeled. Of course, the important sections would be placed more prominently than the others. Dashboard creation is a process of constant refining and experimenting. And in that sense, the overview section would benefit most from constant testing and refining to arrive at the perfect dashboard design. Zoom and Filter Once all the data is presented to the user in the overview section, the viewer will want to focus on particular areas of interest. This involves zooming and filtering the data using the dashboard’s interactive features: zooming, scrolling, panning, drill-down, legend, range selector, etc. For example, zooming may be drilling down from global to country- specific data while filtering may be excluding information in a specific time range. From a design perspective, you should aim to provide the user with plenty of control for zooming and filtering data from the overview. This will yield maximum insights and action from the information at hand. When viewing a dashboard, don’t settle for complex ways to get to the exact data you need. If your dashboard doesn’t support advanced zooming and filtering features, you may want to send in a feature request to whoever created it. After all, zooming and filtering is where the fun starts for a cyber intelligence analyst. Details-on-Demand You’ve identified areas of interest from the overview section, and have dug deeper into the data using zooming and filtering, but you still may not have found what you started looking for. The devil’s in the details! A dashboard that excels at giving an overview, and allows extensive zooming and filtering, should go all the way and give the viewer access to the minute details. This would bring them as close as possible to the raw data, and equip them to find what they started looking for. This third layer of data would be less visual, and more text-heavy with a focus on accurate information rather than trends. This way the analyst gets what he or she needs, in a way that drives action. By using the three steps of the information-seeking mantra, you can avoid information overload, analyze data more easily, and find solutions faster. Let’s look at three examples that follow the information-seeking mantra, and make analysis a lot more interesting. 1. New Relic New Relic has an outstanding network monitoring dashboard. The overview section uses a combination of different chart types – bar, line, and map – to give the viewer maximum information at a glance. It uses a date range selector, drill-down, and interactive legend to allow the viewer to zoom and filter data. One layer deeper, the analyst has the ability to find answers to their questions. Overview 3/7
  9. 9. Details-on-Demand 2. MailChimp 4/7
  10. 10. The second visualization is from MailChimp’s Wavelength product which allows newsletter creators to identify common interests among their newsletter subscribers. It starts with a beautiful, and yet, informative overview, giving the viewer a bird’s eye view of all the connections among their subscribers. The viewer can then zoom into a section that’s densely connected. And finally, they can click on individual points to get the details-on-demand. Overview Details-on-Demand 5/7
  11. 11. 3. Recorded Future Our third example is from Recorded Future, a web intelligence platform that continually scans hundreds of thousands of public web sources. Their system organizes that data for analysis and returns actionable intelligence using six visualization tools. For this article, we’ll focus on a Timeline visualization that’s comparing cyber-related instances for five major corporations. The overview section shows data across a 12-month period in the form of events (colored dots) and references (gray line chart). Once you’ve zoomed and filtered the data, clicking on each dot opens a box with the details-on-demand for each web mention (reference). This dashboard makes it easy to analyze the data at an overview, or granular level, and is an excellent example of how to use the information-seeking mantra. Overview 6/7
  12. 12. Details-on-Demand Additional Reading If you’d like to read Shneiderman’s entire paper on the information-seeking mantra, it’s called “The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations.” Don’t be put off by the long title, it’s actually quite a light read for a research paper. If you haven’t already, please read the previous post on Edward Tufte’s concept of chartjunk, and sparkline charts. And if you like what you’re learning here, stay tuned for my next post on the various chart types used in cyber intelligence. Author Profile: Twain Taylor is a guest blogger for Recorded Future. You can find more insight by Twain about the intersection between data visualization and big data on the FusionCharts blog. 7/7
  13. 13. Give Your Cyber Intelligence Dashboards a Facelift With These Advanced Chart Types The Recorded Future Blog This is the third post in our blog series, “Visualizing Cyber Intelligence.” In the cyber security world, charts and dashboards are indispensable. Yet, few analysts take the time to consider which is the right chart to use for the job at hand. In this post, we look at some commonly used charts, as well as advanced chart types that can turn you into a cyber crime-fighting superhero. This post covers three types of dashboards a cyber intelligence analyst uses on a day-to-day basis: Static Interactive Executive Static Dashboards These dashboards are used to display a very specific section of the overall pool of data to the analyst. Static dashboards focus on past information, and can be generated ad hoc (when needed) through applications, or on a scheduled basis via email. 1/4
  14. 14. It’s good to stick with the basic chart types for static dashboards. These include the bar chart, line chart, and pie chart. Bar charts are great for comparing multiple values, and spotting outliers. Line charts are ideal for doing time- series analysis when you want to spot a trend over a period of time. When using a line chart, one simple yet powerful technique that helps read the data better is to add a trend line. Popular among stock analysts, trend lines make it easy to spot variations in data over time, and identify how quickly a metric rises or falls. Pie charts tend to be commonly used in dashboards. However, as a word of caution, they’re only somewhat helpful if they have up to three to four slices, and become almost impossible to read as the number of slices increase. Though colorful, this pie chart is cluttered, and doesn’t help read the data. With that said, you may want to consider replacing your pie charts with bar charts. You’ll be able to understand your data better, and your dashboard will still look great. Interactive Dashboards Raffael Marty, author of Applied Security Visualization, says the purpose of dashboards in cyber security is “to understand the current state of systems and applications, and presently ongoing tasks or events of interest.” To do this, the security analyst relies not only on static dashboards, but on interactive dashboards that allow slicing and dicing of data. Let’s say you’ve spotted an anomaly in your weekly report, and want to dig deeper to find the root cause, that’s when you’ll switch from your static report to a live application. Static dashboards enable you to spot problems, but 2/4
  15. 15. interactive dashboards give you the information you need to resolve them. Interactive dashboards may contain data as recent as a few minutes ago, or even data that’s updated in real-time. The basic chart types – bar charts and line charts – come in handy here too, but unlike in a static dashboard, they need to be augmented with interactive features like filter, sort, zoom, scroll, tooltips, and more. If you want to drill down to find the exact source of the issue, often, a network chart is the go-to-chart for the cyber intelligence analyst. A good network chart should be able to direct the viewer’s attention to the affected systems, or users immediately, and allow them to follow the nodes to the source of the attack, or issue. Recorded Future has a great example of a network chart that allows viewers to trace the source and context of the issue. It lets you dig deeper by showing a rich tooltip (box) with minute details. Executive Dashboards While cyber security professionals may spend their time firefighting attacks, they always strive to influence their organization’s strategy. This is where executive dashboards come in. Marty, in his book, says CIOs who use executive dashboards often comment, “Show me when my crown jewels are at risk.” CIOs want to see their most important business metrics at a glance whenever they need to. The dashboard acts like a thermometer showing vital health statistics for the entire operation. According to Marty, this gives the CIO “situational awareness” enabling them to “drive decisions and react to, or proactively address, upcoming problems.” 3/4
  16. 16. Here’s an example of an exemplary CIO dashboard created by the leading expert on dashboards, Stephen Few: One of the many reasons this dashboard stands out is for its use of advanced chart types like bullet graphs, and sparkline charts. These charts are rarely used, but are one of the most effective chart types a cyber intelligence dashboard could have. They pack a lot of information in very little space, allowing the viewer to compare data points, and analyze the context in a matter of seconds. As a take away from reading this post, do look at your cyber intelligence dashboards for opportunities to change the chart types, and interactive features in them. This effort will enable you to be much more efficient, and even uncover hidden opportunities from your data. With that we come to the end of our “Visualizing Cyber Intelligence” series. If you enjoyed reading this post, be sure to catch up on the previous two posts as well: The Role of Visualization in Cyber Intelligence How to Use the Information-Seeking Mantra in Cyber Intelligence Dashboards Author Profile: Twain Taylor is a guest blogger for Recorded Future. You can find more insight by Twain about the intersection between data visualization and big data on the FusionCharts blog. 4/4