This presentation covers benefits, use cases, practical examples, potential issues and the approach that needs to be taken when it comes to harnessing the power of dark data (a largely untapped strategic play in the big data realm).
Dark Data Revelation and its
What is dark
The information assets organizations
collect, process and store during
regular business activities, but
generally fail to use for other
- IT Glossary by Gartner
In simple terms, dark data is all that useful data an
organization possesses, but doesn’t actually
meaningfully use or analyze for the improvement of the
The enormous digital universe
2020 44 ZB 37% 27% 10%
4.4 ZB 22% 17% 2%
Total size of
According to IDC (a research firm), up to 90 percent of the
digital universe is unstructured data.
Traditional sources of dark data
Server log files
Networking machine data
Customer queries recorded in calls, emails, forms
Underused employee data
Unstructured information arising out of business mails and presentations
Unused data resulting from business research and surveys
Why is it
Businesses are heavily invested when
it comes to collection of data;
however, tangible value can be
derived only after companies start to
understand their dark data and how
it can be applied.
It is also a sensible step for any company which is getting
started with big data and building a data warehouse.
In this case, dark data can be a reliable source of historical
3 facets of dark data
Data in the deep
Existing unstructured data
Many businesses already have large collections of both structured and
Unstructured data such as emails, notes, messages, documents, logs,
and notifications (including from IoT devices) are confined to the
organization and remain largely unused (due to lack of tools and
techniques or their absence in the database).
These data assets could be potentially having valuable insights related
to competitors, pricing and consumer behavior.
Nontraditional unstructured data
Data present in the web pages, audio and video files and still
images are largely untapped data that can be mined via data
extraction solutions, computer vision, advanced pattern
recognition, and video and sound analytics.
This can help businesses perform advanced analytics on data
present in nontraditional formats to better understand their
customers, employees, operations, and markets.
Data present in the deep web
The deep web presents the largest pool of
unused information—data curated by
academics, consortia, government
agencies, communities, and other third-
Companies can potentially curate competitive intelligence
using a type of emerging search tools developed to help users
target scientific research, activist data, or even hobbyist
threads found in the deep web.
An example of such tool can be Stanford University’s search
engine called Hidden Web Exposer that scrapes the deep web
for information using a task-specific, human-assisted
If the data stored is covered by legal
regulations such as credit card data,
exposure of such data could expose
companies into financial and legal
Companies could intentionally or
unintentionally disclose proprietary
or sensitive data on business
operations, products, financial status
and business plans.
Companies are considered as
protector of data they collect. So, any
loss of data, especially sensitive and
confidential data, can lead to loss of
If a company avoids analysis and
processing of dark data but its competitors
do, then its competitors will be in a better
position to capture more market share by
leveraging the insights from dark data.
Stitch Fix, an online subscription shopping service, uses images from
social media and other sources to track emerging fashion trends and
evolving customer preferences.
Personalization in retail
filled by clients
shipped to the
A financial services firm wanted to gain insight from its trading terminal data to find
correlations between trading patterns and abuses like money laundering and other fraudulent
Most of the data was dark owing to the volume and geographically scattered storage.
After the customer was able to utilize what was previously underutilized, and completed the
data prep and analysis process to determine suspect patterns in transactional records, they
took that analyzed data and created sophisticated predictive models that can identify activities
that indicate the potential for fraud, and take measures to prevent fraud before it occurs.
Instead of attempting to discover and
collect all of the dark data hidden
within and outside your organization,
work with the business team to find
answers for specific business
Getting the right
Source data from the web to
augment your own data with publicly
available demographic, location, and
Being open to
third party data
Data scientists are valuable
resources, especially those who have
the skills to combine deep modeling
and statistical techniques with
industry or function-specific insights.
Advanced visualization software can boost
business intelligence by repackaging big data into
smaller, more meaningful chunks, delivering value
to users much faster.
This is crucial since information can be more
easily consumed when presented as an
infographic, a dashboard, or another type of
Future of dark data
Most of the companies in general will learn to better tap
into their dark data, it’s the way connected and
measurable world is progressing.
The real value will be delivered to those business that
would open their data sources in a secure and
responsible manner within their business so that the
workforce is empowered enough to become problem
solvers in own right.
Reach out to PromptCloud — a pioneer in custom, managed and cloud-based web
https://www.promptcloud.com | firstname.lastname@example.org
Looking to augment data assets with web data?