The document discusses the challenges faced by data-driven organizations in achieving closed-loop processes to enable real-time insights at scale. It outlines how such organizations want all available data in real-time to better inform decisions, but face issues such as limited bandwidth, evolving data formats, and ensuring compliance. The document proposes that a closed-loop "OODA" (Observe, Orient, Decide, Act) system architecture is needed to address these challenges, with distributed regional infrastructures feeding into a low-latency core infrastructure to facilitate real-time command and control and end-to-end data provenance.

1. 1
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
State
of
C4I
:
Cri.cal
Aspects
Computer/Cyber
Technologies
and
Intelligence
The
Data
Driven
Enterprise
Real-­‐Bme
insights
at
scale
through
closed-­‐loop
processes
Joe
WiG

2. 2
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
The
role
of
‘dataflow’
in
a
Data
Driven
Enterprise
Collect
Process
Store/Query
Latency
:
From
sensor
to
repository
(far
less
than
half
the
story)

3. 3
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Enterprise
Dataflow
Different
organiza.ons/business
units
across
different
geographic
loca.ons…

4. 4
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Enterprise
Dataflow
Conduc.ng
business
in
different
legal
and
network
domains…

5. 5
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Enterprise
Dataflow
Opera.ng
on
very
different
infrastructure
(power,
space,
cooling)…

6. 6
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Enterprise
Dataflow
Capable
of
different
volume,
velocity,
bandwidth,
and
latency…

7. 7
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Enterprise
Dataflow
Interac.ng
with
different
business
partners
and
customers

8. 8
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Data
driven
organiza.on
or
leader…
Wants
all
the
data
–
now
To
understand
it
all
and
how
it
relates
To
beGer
inform
decisions
and
decision
makers
To
outpace
adversaries/compeBtors
and
maximize
value
to
customers

9. 9
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Challenges
Wants
all
the
data
–
now
• Bandwidth:
Limited,
Error
prone,
expensive
• Denied
access
• Compliance
requirements
(legal,
policy,
geographic
restricBons)
• CompeBng
prioriBes
(perishable
vs
delay
tolerant)
• Storage
is
cheap
but
rate
of
data
sensing
is
skyrockeBng
• Controlling
the
edge
systems
To
understand
it
all
and
how
it
relates
To
beGer
inform
decisions
and
decision
makers
To
outpace
adversaries/compeBtors
and
maximize
value
to
customers

10. 10
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Challenges
Wants
all
the
data
–
now
To
understand
it
all
and
how
it
relates
• Formats/schemas
evolve
• SemanBcs
of
data
evolve
• Processing
bandwidth
limits
• Signal/noise
results
in
inefficient
processing
• Compliance
implicaBons
of
joining
data
To
beGer
inform
decisions
and
decision
makers
To
outpace
adversaries/compeBtors
and
maximize
value
to
customers

11. 11
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Challenges
Wants
all
the
data
–
now.
To
understand
it
all
and
how
it
relates
To
beGer
inform
decisions
and
decision
makers
• How
to
distribute
accumulated
context
• How
to
track
origin
of
most/least
valuable
sources
• Compliance
implicaBons
of
sharing
observaBons/insights
To
outpace
adversaries/compeBtors
and
maximize
value
to
customers

12. 12
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Challenges
Wants
all
the
data
–
now.
To
understand
it
all
and
how
it
relates
To
beGer
inform
decisions
and
decision
makers
To
outpace
adversaries/compeBtors
and
maximize
value
to
customers
• How
to
maximize
value
of
distributed
enterprise
• Ability
to
redo
decisions
(what
if
I
took
a
different
path)
• Ability
to
interact
with
live
data
and
systems
to
effect
change
• Trace
acBons
to
decisions
to
inputs…

13. 13
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
‘OODA
Loop’
–
A
closed
loop
system
Wants
all
the
data
–
now
OBSERVE
To
understand
it
all
and
how
it
relates
ORIENT
To
beGer
inform
decisions
and
decision
makers
DECIDE
To
outpace
adversaries/compeBtors
and
maximize
value
to
customers
ACT

14. Page 14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
It’s not just how quickly you
move data – it’s about how
quickly you can change behavior
and seize new opportunities

15. 15
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
The
Data
Driven
Enterprise
–
demands
a
closed
loop
architecture
§ Constrained
§ High-­‐latency
§ Localized
context
§ Hybrid
–
cloud/on-­‐premises
§ Low-­‐latency
§ Global
context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

16. 16
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Command
and
Control
§ Constrained
§ High-­‐latency
§ Localized
context
§ Hybrid
–
cloud/on-­‐premises
§ Low-­‐latency
§ Global
context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE
Distributed
agent
Central
control
Mul.-­‐tenant
Interac.ve
command
and
control

17. 17
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Provenance
§ Constrained
§ High-­‐latency
§ Localized
context
§ Hybrid
–
cloud/on-­‐premises
§ Low-­‐latency
§ Global
context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE
Origin
–
a[ribu.on
Replay
–
recovery
Evolu.on
of
topologies
Long
reten.on
Types
of
Lineage
• Event
(runBme)
• ConfiguraBon
(design
Bme)

18. 18
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
The
need
for
data
provenance
For Operators
• Traceability, lineage
• Recovery and replay
For Compliance
• Audit trail
• Remediation
For Business / Mission
• Value sources
• Value IT investment
BEGIN
END
LINEAGE

19. 19
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Security
§ Constrained
§ High-­‐latency
§ Localized
context
§ Hybrid
–
cloud/on-­‐premises
§ Low-­‐latency
§ Global
context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE
Sign,
encrypt,
sta.c
(data
and
control)
TLS,
obfusca.on,
dynamic
en.tlements
Kerberos,
PKI,
AD/DS,
etc.

20. 20
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
The
need
for
fine-­‐grained
security
and
compliance
It’s not enough to say you have
encrypted communications
• Enterprise authorization
services –entitlements
change often
• People and systems with
different roles require
difference access levels
• Tagged/classified data

21. 21
©
Hortonworks
Inc.
2011
–
2016.
All
Rights
Reserved
Topics
to
explore
further
Ã Private
InformaBon
Retrieval
(PIR)
–
Doesn’t
this
break
the
provenance
trail?
Ã Role
of
cryptographically
verifiable
approaches
such
as
Blockchain
Ã What
standards
are
needed
(provenance,
security)?