15 events to log for better dev ops

15 EVENTS TO LOG
FOR BETTER DEVOPS

| Rapid7.com 215 Events to Log for Better DevOps
TABLE OF CONTENTS
3
5
6
7
8
9
10
11
12
14
15
16
17
18
19
20
21
22
23
24
25
26
Foreword ...............................................................................................................................................................................................
Better DevOps for the Sales Team .............................................................................................................................................
1. Logging your most common sales CRM errors ..........................................................................................................................
2. Monitoring the rate of transactions from your billing platform ..............................................................................................
3. Tracking discount codes created in your e-commerce system ...............................................................................................
Better DevOps for Customer Success .......................................................................................................................................
4. Tracking VIP Support requests ....................................................................................................................................................
5. Tracking days since last login ......................................................................................................................................................
6. Tracking user session lengths ......................................................................................................................................................
Better DevOps for the Product Team ......................................................................................................................................
7. Tracking most clicked features ....................................................................................................................................................
8. Tracking the usage of a feature over time .................................................................................................................................
9. Tracking feature requests and feature releases ........................................................................................................................
Better DevOps for the Marketing Team ..................................................................................................................................
10. Tracking traffic from a source ....................................................................................................................................................
11. Tracking the performance of advertising mediums .................................................................................................................
12. Locating 302 Redirects ................................................................................................................................................................
Better DevOps for the Data Science Team ...........................................................................................................................
13. Identifying slow Tableau workbooks ........................................................................................................................................
14. Monitoring failed Tableau extracts ...........................................................................................................................................
15. Monitoring Tableau Apache 404s .............................................................................................................................................
Conclusion ............................................................................................................................................................................................

INTRODUCTION
Despite the wave of discussion around DevOps that has emerged over the past couple
years, the majority of companies today are still far from fully embracing it. Puppet
Labs’ 2015 State of DevOps Report1
reveals vast differences between companies who
practice DevOps and those who don’t, with DevOps-focused organizations reporting
“60 times fewer failures and recovery from failure being 168 times faster than their
lower-performing peers”. With such dramatic benefits, it’s surprising to still see so many
organizations not embracing a DevOps culture.
Perhaps just as surprising though, is the overwhelming focus of DevOps on IT
performance. Of course the emergence of complex and highly distributed IT
infrastructures created the need for planning and process automation that DevOps grew
to satisfy. Yet DevOps seems to have plateaued at IT performance, with little written to
date on how the principles of DevOps are being applied outside of IT’s own department.
In Gene Kim’s The Phoenix Project, (a short story applauded for summarizing IT’s
modern challenges and making them relatable to a non-technical audience), Kim’s
protagonist regularly interacts with executives from a variety of departments, including
Engineering, Operations, Marketing, Security, Sales, Finance, and others, providing an
idealistic example by the end of the story of how IT operates as the backbone of the
organization. Yet when talking about DevOps in the real world, the conversation is almost
always entirely focused on Engineering and Operations. “IT Automation”, “Continuous
Integration”, “Infrastructure as Code” and other buzz phrases are commonly used to
describe DevOps and all stop short of extending beyond the walls of the engineering and
operations departments. If IT is truly the backbone of any good organization (even those
outside of the technology industry), where is DevOps being practiced across the rest of
the company?
1
https://puppetlabs.com/2015-devops-report

There are a variety of factors that make up “good DevOps”, not all of which are necessarily
applicable to other departments. For example, as far as Marketing has come in embracing
data analytics, an “Infrastructure as Code” mentality doesn’t yet apply to Marketing’s
needs. “Planning” and “Monitoring” are two practices that do fit non-technical teams
though and warrant serious consideration. Many of the tools being used by Finance,
Marketing, and Sales teams today are set up by the IT department. Many of the functions
these tools serve have a very real impact on business performance, whether it be a sales
CRM, an e-commerce portal, or a marketing analytics platform that directly influences
how budgets are spent. When these tools and functions break down, progress towards
business goals are immediately interrupted and likely result in urgent calls to an IT team.
Often, the problem was discovered by a frustrated customer or a frantic employee
noticing that online sales have dropped to zero for the past hour. And even more often,
there’s no defined plan for remediation. Other IT work must be put on hold while dealing
with unplanned fire drills.
In an effort to further reduce unplanned work and empower teams through good DevOps
practices that permeate the entire organization, collaborative planning and automated
monitoring practices should be happening in every department. Throughout this paper,
we’ll explore each department of a typical organization, covering some of the more
common challenges these departments face and how good DevOps can be applied in
each case.
Real Examples
In the following sections, we’ll explore how log data from a wide variety of tools and
services can be used to practice good DevOps and support an entire organization. For
demonstration purposes, we use Rapid7’s infrastructure monitoring tool, InsightOps and
LEQL (The Rapid7 Log Entries Query Language). For more information on these tools,
visit Rapid7.com.

BETTER DEVOPS
FOR THE SALES TEAM
For many organizations, the sales team is the group who keeps the
lights on. A sales team’s actions correlate directly to revenue for the
business and every minute a salesperson is unable to do their job
results in lost or delayed revenue. The modern sales team leverages
a wide spectrum of tools from e-commerce platforms, customer
relationship management tools and payment gateways. These vital
systems generate a wealth of log data that can be easily captured.
Bringing the power of DevOps to sales can improve the reliability,
efficiency, and security of these essential tools.

Customer relationship management tools are at the core of many businesses today. These
platforms typically generate a wealth of log data. Despite the importance of these systems, it is
surprising how infrequently the logs created by CRM tools are reviewed.
The example query below utilizes groupby() and sort() functions to return the Salesforce
errors that are occurring most frequently.
Source: Logentries.com
Example query:
1. Logging your most common sales
CRM errors
where(severity = ERROR AND /message=.(?Pregmessage[^:(“]*)/) groupby(regmessage) sort(desc)
Try with InsightOps

Your organization is no doubt tracking the transactions flowing through your billing platform
in terms of monetary impact. Surprisingly, it seems less common to monitor the number of
transactions occurring and the rate of which those transactions occur, disregarding (for this
metric) the value of each transaction.
The example visualization below is returned when using a calculate(count:x) function. This
shows the average number of transactions over time, quickly revealing a drastic drop. This drop
could trigger an alert and provides context around failure timeframe.
Source: Rapid7.com
2. Monitoring the rate of transactions
from your billing platform
Example query:
where(transactionId) calculate(count) timeslice(24)
Try with InsightOps

Tracking your average discount issued over time is helpful when considering financials. The
security perspective also provides an interesting use case for tracking discounts. In 2015, The
Check Point Malware Vulnerability Research Group discovered a remote code execution
vulnerability in Magento that, among other things, would allow an attacker to create a 100%
discount code3
.
The example below uses a calculate(average:x) function to examine the trend of discount
codes created in our simulation environment. The large spike highlighted, occurred when a
someone successfully applied a 100% discount code.
3
http://blog.checkpoint.com/2015/04/20/analyzing-magento-vulnerability
Source: Rapid7.com
Example query:
where(discount) calculate(average:discountValue)
3. Tracking discount codes created
in your e-commerce system
Try with InsightOps

BETTER DEVOPS FOR
CUSTOMER SUCCESS
Customer Success is imperative to the continued operation of an
organization. Protecting the revenue of the company by keeping
churn as low as possible is the primary objective of this team.
Reliable metrics to determine customer satisfaction and outreach
effectiveness are necessary for this integral team to ensure
customer retention. Effectively monitoring your product and
integrating with tools like support helpdesks can provide immensely
helpful information to measure customer engagement, health and
ultimately reduce churn.

Your Customer Success Team likely spends significant time working in and reporting off of a
helpdesk tool. These systems can typically perform standard alerting and escalation as required.
However, configuring alerts beyond the provided scenarios can be difficult or impossible. By
configuring your helpdesk tool to record events in your infrastructure monitoring tool, you open
the possibility for extremely customizable and robust alerting. By monitoring events taking place
in the helpdesk, the Customer Success Team can be empowered to perform more proactive
outreach.
The example below is a basic alert configuration in InsightOps. The pattern used here will be
matched when requests are submitted to the helpdesk by Acme Corp. The alert is configured to
trigger if Acme Corp. submits two support requests in a one-day period, which may suggest an
ongoing and unresolved issue.
Source: Rapid7.com
4. Tracking VIP support requests

Customers who have not logged into your application for an extended period of time are ideal
for proactive outreach. A user may not be using the product because of a technical blocker, the
need for onboarding or a lack of training. Metrics like “days since last login” can be integral to the
operation of a Customer Success Team.
The example below uses the a calculate(unique:x) function to count the number of
customers who have not logged in for over two weeks.
Now you have a high-level metric representing the number of customers who have not logged
in for a considerable period. This is helpful to watch over time and evaluate if blanket campaigns
and efforts are effective. However, the Customer Success Team will also want to know the
individual customers so that they can reach out.
The example below displays results grouped by customer, providing the Customer Success team
with a list of specific individuals to contact.
Source: Rapid7.com
5. Tracking days since last login
Example query:
where(daysSinceLogin 14) calculate(unique:customer)
Try with InsightOps

User session lengths in your application can be a strong indicator of customer health. A direct
correlation is often exhibited between increased session lengths, customer satisfaction and
product adoption.
Monitoring application session lengths for your Customer Success Team provides a meaningful
metric to evaluate the effectiveness of campaigns and onboarding efforts. An average session
length in decline can be a sign of an underlying issue.
The example below uses a calculate(average:x) function to examine the average session
length for all customers in seconds.
Source: Rapid7.com
6. Tracking user session lengths
Example query:
calculate(average:sessionLength)
Try with InsightOps

The Customer Success Team could also review the session length for a single customer to
determine the effectiveness of their recent targeted efforts.
Source: Rapid7.com
Example query:
where(customerId = 123) calculate(average:sessionLength)
Try with InsightOps

BETTER DEVOPS FOR
THE PRODUCT TEAM
The product team is where the business and engineering aspects
of the organization meet. This team works to understand the needs
of the customer to design and improve the product. The product
team is continuously working with customers and other members of
the organization to understand how the product can be improved.
However, unless a customer purposefully provides feedback, it can
be challenging for product teams to know how an application is
being used or what changes should be implemented. Client side
logging (also known as application layer logging) provides insightful
data for the product team. Tracking clicks, sessions and other user
behavior permits the Product Team to make informed decisions
about the current product and the product roadmap.

Using an application language library such as the InsightOps node.js, .Net or Java libraries allows
you to log from the front end of your application and get in-depth information about user
behaviors.
Tracking clicks on certain features or buttons allows you to perform analysis on how and which
features are being used.
The example query below uses groupby() and sort() functions to show the most clicked
features.
Source: Rapid7.com
7. Tracking most clicked features
Example query:
where(clicked) groupby(feature) calculate(count) sort(desc)
Try with InsightOps

Source: Rapid7.com
Similar to the example from number 2, a drastic drop in the use of a feature can be a sign of
application or even system failure. A prolonged drop in the use of a feature also plays a helpful
role in product team investigations.
The example below shows the count of clicks of a feature during an infrastructure failure. The
visualization is produced using a calculate(count) function.
8. Tracking the usage of a feature
over time
Example query:
where(clicked AND viewerFeature) calculate(count)
Try with InsightOps

At a glance, a visual dashboard of log data can show you feature releases compared to customer
requests, providing exceptional perspective.
The dashboard widget below is being populated with log data from a Trello board. As cards are added
to the ‘Suggested Feature’ and ‘Released Feature’ columns, the events are recorded in the logs.
The graph below uses two identical queries, except for the Trello column name, using a
calculate(count) function to populate the graph.
Source: Rapid7.com
9. Tracking feature requests and
feature releases

BETTER DEVOPS
FOR THE
MARKETING TEAM
Marketing is the lifeblood of every company. This team generates
brand awareness and brings in new customers to kick off the sales
cycle. Today’s marketing team pursues its goals through many
mediums, especially online promotion. The marketing team will
almost always have a traffic analysis tool like Google Analytics to
monitor the visitors they capture. However, log files can provide
additional context to the data not provided by such a tool. The
log files afford a different perspective, exact results and real time
information. This paper focuses exclusively on monitoring web
server logs (such as Apache and Nginx) for the marketing team.
However, like all departments, the marketing team can also benefit
from monitoring numerous systems such as content management,
CRMs and other tools.

The exactness of web requests and timestamps provided by log data can enable marketing
teams to act quickly and accurately to changing conditions and issues.
It can sometimes be challenging to spot a dramatic drop in site traffic from a specific source if
overall traffic hasn’t experienced a dramatic dip. Using web server logs, marketing teams can
visualize traffic from unique sources and establish advanced inactivity or anomaly alerts to
better monitor web traffic.
The query below uses a calculate(count) function to search Nginx access logs for traffic over
time from a particular partner.
Source: Rapid7.com
10. Tracking traffic from a source
Example query:
where(/dzone/) calculate(count)
Try with InsightOps

Source: Rapid7.com
Source: Rapid7.com
11. Tracking the performance
of advertising mediums
Example query:
where(/utm_medium=cpc/)
The combination of tracking parameters added to a URL by the marketing team and the
access logs from your web servers can provide powerful and immediate intelligence into the
performance of various mediums.
Using the simple search below, Marketing teams can monitor the traffic sent from a particular medium.
Add a calculate(count) function to view the traffic from the selected medium over time.
Try with InsightOps

Your marketing department works hard to create and promote high quality content to
continuously improve search engine ranking. Major potential gains in SEO may be lost due to
seemingly small issues, such as the use of 302 redirects.
When a 302 redirect is used on a page, none of the rankings nor trust established by the
indexing search engine is passed to the destination page, which can harm your site’s search
engine rank. A conscious effort should be made to weed out 302 redirects and replace them
with 301 redirects whenever appropriate.
Your marketing department may be affected by this redirect issue without even knowing. A
simple query of your web server logs can return all of the pages being served with a 302 status.
The query below uses a groupby() function to search Nginx access logs to return the names of
any pages producing a 302 status.
Source: Rapid7.com
Example query:
where(status = 302) groupby(path)
12. Locating 302 Redirects
Try with InsightOps

BETTER DEVOPS
FOR THE DATA
SCIENCE TEAM
Data science is starting to play a major role in the operation of
many organizations. The DataOps professionals crunching this data
engage numerous tools, with one popular choice being Tableau.
Capturing the logs produced by Tableau infrastructure allows for the
real-time monitoring of events, deeper data analysis and efficient
troubleshooting of issues.

Slow Tableau workbooks can be extremely frustrating for both that workbook’s users and other
users of the same Tableau infrastructure. Monitoring the VizQL log can allow you to identify
problem workbooks in the system.
Here is an example of what you can expect to find from a VizQL log:
The example below shows the workbooks with the highest average load time. This visualization
is produced by using groupby(), calculate(average:x) and sort() functions.
13. Identifying slow Tableau
workbooks
13 Oct 2015 16:36:17:810 {
“ts”: the timestamp for this record
“pid”: the process id for this record
“tid”: the thread id for this record
“sev”: severity
“req”: the http request id
“sess”: the session id
“site”: the site name for this record
“user”: the user name
“k”: the type (“key”) of the record
“v”: {
the payload
}
}
Example query:
where(k=”end-update-sheet”) groupby(site) calculate(average:elapsed) sort(desc)
Try with InsightOps

When Tableau extracts fail, errors are logged to the Background Log. This log includes both
errors and information about other activities and subscriptions within the Tableau system, such
as the logs below:
When trying to investigate a failed Tableau extract, you may want to filter out all INFO level
events so you can focus on events relating to errors. The simple query below uses a regular
expression query to match all log events that don’t contain the word “info” (case insensitive).
Source: Rapid7.com
14. Monitoring failed Tableau extracts
Example query:
where(-/info/i)
Try with InsightOps

The Tableau web service is run by a standard Apache server. The Apache server produces
typical access and error log events, including many of the same web server metrics discussed
throughout this paper.
The Apache server operating as part of your Tableau infrastructure can reveal request errors
and where requests are coming from.
The query below uses the a groupby() function for searching Apache access logs to return the
names of any pages producing a 404 status; revealing that a workbook has potentially been
moved or accidentally deleted.
Source: tableau.com/404
15. Monitoring Tableau Apache 404s
Example query:
where(status=404) groupby(path)
Try with InsightOps

CONCLUSION
Good DevOps should be applied throughout the organization, not just in the IT
department. The monitoring practices suggested here do not have a focus on system
performance but instead focus heavily on achieving the goals of the business. Some of
the metrics discussed monitor and improve system reliability while others are intended
to help make all teams in the organization better informed. The application of DevOps
principals throughout the organization brings this improved reliability and enhanced
knowledge together to result in an organization that is collaborative, productive and
ultimately successful.

ABOUT RAPID7
With Rapid7 (NASDAQ: RPD), security and IT professionals gain the clarity and confidence
they need to protect against risk and drive innovation. Rapid7 analytics transform data
into answers, eliminating blind spots and giving customers the insight they need to
securely develop and operate today’s sophisticated IT infrastructures, networks, and
applications. Rapid7 solutions include vulnerability management, penetration testing,
application security, incident detection and response, SIEM and log management, and
offers managed and consulting services across its portfolio. Rapid7 is trusted by more than
6,200 organizations across over 110 countries, including 38% of the Fortune 1000. To learn
more about Rapid7 or get involved in our threat research, visit www.rapid7.com.
ABOUT INSIGHTOPS
InsightOps automatically combines live log and asset data from across your infrastructure
into one central and searchable location, so you can access the insight you need, when
you need it. Stop guessing where to start your next IT investigation—get InsightOps and
get answers. Learn More at rapid7.com/insightops.

15 events to log for better dev ops

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to 15 events to log for better dev ops

Similar to 15 events to log for better dev ops (20)

More from Abhishek Sood

More from Abhishek Sood (20)

Recently uploaded

Recently uploaded (20)

15 events to log for better dev ops