1. Become a Data-
driven Company
Your Guide to Going Beyond Business
Intelligence Now and into the Future
WHITE PAPER

2. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 2
The data-driven business model
is more than a buzzword, it’s the
reality of competitive, business
intelligence as we know it.
This white paper provides an in-
depth overview of past, present,
and future data systems. From
traditional data models that rely
on Excel, to the future of data
science with Apache Spark and
Natural Language Processing,
we take a deep dive into what
constitutes a true data-driven
organization.
ABSTRACT

3. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 3
The Dilemma
YOUR DATA SYSTEMS GREW ORGANICALLY.
Now when you want to understand how your business works, you have a lot of work to do. You
need a more efficient way of doing business. You need to drive action across your organization
that isn’t based on “gut” feeling.
HOW DID YOU GET HERE?
Usually you were looking to solve a series of immediate problems. You had a sales team, you
needed a CRM. You needed to track financial data, you bought an accounting system. You have
vendors with their own datasets and feeds. Now when you want to understand how that data
fits together you have to manually export and aggregate each individual report.
THE TRADITIONAL WAY TO ACCESS YOUR DATA WAS TO QUERY EACH
SYSTEM BY HAND, create a report, and send that around so people could make
decisions. Those decisions are based on individual knowledge and experience, and your
organically grown process.
LIMITATIONS OF SPARK STANDALONE CLUSTERS
While running Spark in standalone mode is fine for experimentation and PoCs, any serious
deployment of Spark applications is going to be limited by the standalone cluster manager’s
first-in-first-out (FIFO) scheduler, and the default behavior of allocating the entire cluster’s
resources to an application. For instance, a long-running ETL application could grab hold of
all the available resources and prevent any other applications from running. By using YARN
or Mesos, resources will be used more efficiently (especially if dynamic allocation is used).
YOU WANT A SYSTEM THAT SHOWS YOU THE BUSINESS AT A GLANCE,
drills down into the specifics, and gives you confidence in the data.
YOU NEED A SYSTEM THAT HELPS YOU MAKE DATA-DRIVEN DECISIONS
through defined processes.
FINALLY, YOU NEED A SYSTEM THAT IS ROBUST AND ADAPTABLE
to your business as it grows and changes.
The Evolution of Business Intelligence
In order to evolve
from an antiquated
infrastructure,
organizations
must adopt a
more centralized
approach to their
data architecture
and analysis.
DATA CONSOLIDATION
& ANALYTICS
COMPUTER-AIDED
DECISION-MAKING
REAL-TIME COMPUTER
AIDED DECISION-MAKING
SEMANTIC WEB,
NATURAL LANGUAGE,
EVERYTHING-AS-A-SERVICE

4. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 4
Traditional, Present & Future Systems
20TH CENTURY DATA SYSTEMS
It starts with internal data sources, third party data sources, and Excel. To assemble a holistic
view of the company’s status and monthly projection, someone needs to do manual queries.
The results need to be manually entered into an Excel document, where you can manually
produce graphs. These graphs produce
emails, the emails are distributed, and
people make decisions. These decisions
are made based on the knowledge and
experience of the decision-makers and
the organically grown process. While
the process seems laborious and time
consuming, many companies still live
here.
TURN OF THE CENTRUY
DATA SYSTEMS
In order to evolve from the antiquated model above, organizations must adopt a more
centralized approach to their data architecture and analysis. The first step of analyzing a
business is to consolidate all of the major sources of data. Upgrading the human task of
generating Excel spreadsheets into an automated, system-driven process allows for a more
efficient and standardized analysis of a company’s data. Many companies deployed expensive
proprietary Enterprise Data Warehouse (EDW) systems like Teradata and Netezza in order
to become more efficient.
However, this technology was
out of the price range for
much of the midmarket, which
attempted to do similar things
using transactional databases
like Oracle and SQL Server.
However, these systems do not
scale well, make it difficult to
manage change, and require a
lot of resources to deliver the
functionality comparable to
Enterprise Data Warehouses.
Now, newer so-called Big Data technologies like Impala, Hive, and Hadoop, originally
developed for really big Internet companies like Yahoo, Google, Amazon, and Facebook,
have brought the price down for similar functionality. As a bonus these technologies allow
companies to either deploy “on-premises” or in the Cloud.
Meanwhile, Tableau has brought attractive, mainstream analytics to the mid-market, including
a “self-service,” point-and-click interface. While large companies have deployed more
complicated tools like Business Objects, the mid-market has embraced tools like Tableau that
work with “Big Data” technologies from a convenient desktop interface.
REO
CELINK
RMS
FHA
SALESFORCE
SHORETEL MORTGAGE
CADENCE
VELOCIFY
WEB
ANALYTICS
DATA
LAKE
REO
CELINK
RMS
FHA
SALESFORCE SHORETEL MORTGAGE
CADENCE
VELOCIFY
WEB
ANALYTICS
20TH CENTURY DATA SYSTEM
TURN OF THE CENTURY DATA SYSTEM

5. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 5
Ultimately this provides a convenient a holistic view
of the company, comprehensive visualizations and
analytics, and overall better decision making.
EVOLVED DATA SYSTEMS
The next step in evolution of the data-driven business
model is to codify and automate your business
processes and decision-making.
In 20th and turn of the century data systems we mainly see the same process humans have
followed in decision-making since the beginning of time. Data comes in, a person analyzes it
and then makes a decision based on their own personal knowledge and experience. Studies
show us that algorithms make better decisions than “gut” instinct, yet most business is
powered by the executive and middle management “gut”.
To do this requires a mindset change that’s equally as important as the shift in technology. Data
and decisions are the point of business processes, not the side effects. Business processes are
diagrammed, documented, and codified rather than predominantly transmitted by tradition.
The company’s culture and people being empowered to do the right things still matter
-- perhaps even more. The mindset is whether you see yourself as a circuit in the machinery
of the day-to-day business process, or whether you see yourself as someone managing and
auditing the business process.
In companies with evolved data systems, data and actions come in and are run through a
computer-
codified business
process and a
set of rules for
how decisions are
made. The easily-
interpretable
data (orders,
invoices, etc.) are
directly reacted
to by the system
and tabulated
for periodic
reporting. The
less easily-interpretable data (natural language customer service requests) are automated as
much as currently possible, and then routed to the appropriate person for further help.
Changes to the company’s procedures are managed and implemented through a governance
process that ensures change is rolled out consistently. This may still require some staff
retraining, but with data systems driving the process, its global adoption is easier than
directing each person involved in implementing a process.
The main role of a person in automated companies is to look at the data in new ways and find
new opportunities. Rather than use creativity in handling day-to-day tasks, they find
REO
CELINK
RMS
FHA
SALESFORCE
SHORETEL MORTGAGE
CADENCE
VELOCIFY
WEB
ANALYTICS
DATA
LAKE
BPM
RULES
ENGINE
EVOLVED DATA SYSTEM

6. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 6
the exceptions and help invent new processes as well as evolve older ones to the changing
business environment. Sometimes this is a few simple parameter changes. Sometimes this is a
significant change in the codified process. Generally, procedural changes do not require major
engineering efforts, they require more of a creative approach to design and analysis.
Although the alteration in management required for full implementation of this model
requires a considerable amount of time, and some major companies are already undergoing
this transition, especially in the more competitive areas of financial services. Although
the technology and expertise is currently available to the mid-market, we have yet to see
widespread adoption.
FUTURE DATA SYSTEMS
Modern data systems still mainly process data in batch. The next stage is to move to “real-time”
technologies and make the entire company operate on an “event” instead of the period.
We have become accustomed to batch processing by the year, the quarter, and the month.
So have your suppliers (Your supply order for the quarter is...) as well as your employees and
customers. However, the world doesn’t really work this way. Recently, the market in China
underwent a rapid transformation. Any company that had significant plans that focused on
that region for the year or quarter had to react quickly. Customers change their minds, world
events occur, and nothing happens on your schedule. Everything from money laundering, fraud
detection, and promotions are generally handled in batch. However, there is another way of
working. As events happen, you tally them up, and once they go over a threshold you make a
decision. With modern data systems we don’t actually have to batch data to make decisions, we
can tally and operate on thresholds.
Newer businesses have
been moving to “real-
time” in order to disrupt
older ones. Dell famously
had little inventory and
made computers to order.
Amazon has disrupted
the entire retail industry
through a distribution
system that takes orders
and puts inventory at the
customer’s door in two
days or less. In both retail and manufacturing “Just-In-Time” is a well-sought goal.
The world of financial services is moving to real-time faster than almost any other industry. In
this sector you often have no tangible asset that requires complex logistics. This makes moving
to an event-based system simpler. In other industries it requires renegotiating your relationship
with your vendors and suppliers to provide goods and services “on-demand,” and to be able to
adjust your orders or labor up or down as the market or events demand. Moving to real-time
also requires a rethink of older IT infrastructure and a reduction in its complexity.
By adopting this model, reporting over large datasets goes away for day-to-day operations.
Companies that
are real-time and
automated will react
faster to the market
and can evolve as
change occurs.
Ultimately, this is
the competitive
advantage.
REAL-TIME
STREAMS
PROCESS RULES
CUSTOMER
SUPPLIER/VENDOR
SALESPERSON
STORAGE DATA STORE
FUTURE DATA SYSTEM

7. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 7
Processing large datasets is still necessary for historical analysis, but most data is immediately
up to date. Companies that are real-time and automated will react faster to the market and can
evolve as change occurs. Ultimately, this is the competitive advantage.
Achieving a Modern Data Architecture
& Going Beyond
CONSOLIDATION
It is difficult to be data-driven if you don’t have a holistic view of your data. Moreover, the
agents of this change are not the systems but the people who make the company work. They
need to be able to efficiently and effectively use the data. The only way to do that is to bring
data together.
This requires creating an inventory of data assets, a central repository (aka data lake,
enterprise data hub), views of the data, and then mapping to business roles for security
purposes. Ideally, both data governance tools and processes are put into place at the same
time.
From a technology perspective you can use tools like Sqoop and Kettle to feed the data to
a Hadoop-based data repository (Impala/Hive). These feeds are scheduled with Oozie or a
similar tool. This process is best done in combination with the analytics process, although
without clear use cases this process will go out of scope and fail.
ANALYTICS
The goal is to make data “self-service,” so when someone has an idea, they can go directly to
the repository for the data, rather than having to ping IT or the department sourcing the data.
Getting to self-service, however, is no small matter: it requires structuring the data.
In general, the process is as follows: views
are created for cross-department or
cross-datasource reports and analytics.
Initial sets of dashboards and reports
are created in newer, more attractive
formats. An analytics tool is purchased and
deployed throughout the organization,
and the staff is trained on its use and
function.
Both the users and the providers of the data are interviewed in order to get a complete picture
of needs, abilities, and processes which drive report creation using
tools like Tableau. This work is best done concurrently with the data
consolidation as both processes require use cases.
PROCESS MAPPING & AUTOMATION
Automating a process requires understanding the process.
This involves reviewing the company’s source information and
interviewing the executives and decision-makers. However, it also

8. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 8
requires interviewing the people who actually run the process on a day-to-day basis. The
informal process is perhaps even more important than the formal. The output from this
activity is process mapping which usually takes the form of a diagram. With many Business
Process Management systems, the tool that is used to generate a diagram also creates a
runnable configuration file for a business process automation tool. After the initial diagram or
diagrams are created, a data systems approach is used to plug in computational processes.
This is both human analysis and systems analysis, and requires some transformation of both.
Simply implementing a tool such as JBPM isn’t sufficient; real change is required in how the
organization operates.
DECISION MAPPING & AUTOMATION
Process and decision mapping are
best done in parallel. The idea is to
map things that are algorithmic or
numbers-based. Some of these take
place informally, meaning someone
may look at a bar chart, see if two
bars are about the same length,
remember that business tends
to pick up after the summer and
decide to place an order. However, this is an informal system that can be replaced by formal
rules. Rules like this can have adjustable thresholds and parameters.
There are multiple ways to accomplish this, from writing rules in a “rules language” such as
Red Hat’s JBoss Rules, IBM’s WebSphere ILog Rules, to creating a Domain Specific Language
and expressing those rules or using decision tables. There are places where actual algorithms
might be implemented using R or Python and possibly Spark for in-memory execution at scale.
This activity works best if started just after the initiation of the business process mapping; the
two are interrelated, and the output of decisions frequently affects process.
GOVERNANCE
Everything can change. The data, the processes and parameters, the decisions, even the rules
or algorithms for making decisions. A system needs to be in place to govern the data and
establish its source, validity, and to manage its structure. Having a centralized data lake isn’t
helpful if you look at a field in a table and can’t answer “What is this? Where did it come from?
What does it mean?”
Processes need explanations and stories as a way to suggest, implement, and review changes.
They also need to be periodically reviewed to ensure they are not stale or out of touch.
Decisions and their parameters (along with any associated algorithms) need a similar review
and change control process. In the meantime, executives may change strategies or add new
lines of business.
Success depends on having an efficient way to adapt ever-evolving systems. For data there are
tools like Hadoop Revealed or Collibra. Process and rules change tools, however, are currently
subpar, but common software revision control systems like git or SVN can provide assistance.
The most important piece is getting that data governance system in place.
Having a centralized
data lake isn’t helpful
if you look at a field
in a table and can’t
answer “What is
this? Where did it
come from? What
does it mean?”

9. T H E L E A D E R I N B I G DATA C O N S U LT I N G | M A M M OT H DATA .C O M 9
ROME WASN’T BUILT IN A DAY
If you’re at the hand-made spreadsheet stage, don’t try to become a real-time, data-driven
company in one project. Cultural change is just as fundamental as the evolution of technology
and eventual adoption. Situations and events that cannot be handled by a core automated
process should not be neglected -- instead, address situations requiring human attention, but
consider how the process might be adapted. You may find that certain events are not “one
offs,” but that multiple events are percolating throughout the company without the information
being shared.
ENGAGING MAMMOTH
Mammoth Data can lead you through every step of this transition, from constructing the data
lake, to helping you create the analytics, to business processes and beyond. We specialize in
Big Data technologies like Spark and Hadoop, as well as cloud-ready databases like Couchbase
and MongoDB. Our data scientists can also offer fresh and innovative insights into your data
and business model. We understand the vision and goals of becoming data-driven, and can
help you get there by delivering industry-leading knowledge and expertise.

10. ABOUT US
Mammoth Data is a Big Data consulting firm specializing in Hadoop®, NoSQL
databases, and designing modern data architectures that enable companies
to become data-driven. By combining cutting-edge technologies with a high-
level strategy, we are able to craft systems that capture, organize and turn
unstructured information into real business intelligence.
Mammoth Data was founded as Open Software Integrators in 2008 by open
source software developer, evangelist and now president, Andrew C. Oliver.
Mammoth Data is headquartered in downtown Durham, North Carolina.
(919) 321-0119
info@mammothdata.com
@mammothdataco
mammothdata.com
MAIN OFFICE
345 W. MAIN ST. SUITE 201
DURHAM, NC 27701