The document discusses the complexities of streaming analytics and data collection systems within organizations, highlighting the reliance on both live and persisted data. It emphasizes the need for streaming infrastructure that enables the reuse of data across various applications and demonstrates the challenges posed by existing information systems. Third Nature, the firm behind the commentary, aims to assist companies in leveraging analytics and information strategies to improve their operational effectiveness.

1. 1
Live Wire: The Next
Dimension for
Streaming Analytics
Alternatives for collecting
and using streaming data
Analyst commentary
December, 2015
Mark Madsen
Third Nature
@markmadsen

2. Copyright Third Nature, Inc.
Almost all data today flows on networks, mostly
used after collection, much of it never stored
2
Source: Noumenal
Disconnected
Milliseconds Minutes Hours+

3. Copyright Third Nature, Inc.
Source: Noumenal
Milliseconds Minutes Hours+
Events and monitoring occur in different time
frames, follow different cycles of use
3
Disconnected
Data lives in multiple places,
at multiple levels of detail, for
differing durations. Unlikely to
all be in one place.
Nor should it be.

4. Copyright Third Nature, Inc.
Streaming infrastructure, collection systems, big
data, tend to be viewed as standalone systems
Most talk about this topic
focuses on streams and
programs living on the
network.
Usually viewed as
independent from the rest of
data use in the organization.
The problem is that the live
and persisted data are used
together, or have
dependencies of use.

5. Copyright Third Nature, Inc.
Monthly
Production plans
Weekly pre-
orders for
bulk cheese
Availability
confirmation
and location
In store system
Store
Stock
Management
Store EPOS
data
Category
Supervis
or
Stock
adjustments/
order
interventions
Order
adjustment
Stock/order
interventions
*
*
Orders
(based on 6
day
forecast)
Dallas
Distrib Centre
WMS
Picking/load
teams
Pos/Pick
lists/Load
sheets
Confirmed
Deliveries/
Confirmed
picks +
loads
Farmers
Milk intake/
silos Cheese plant
Plant
Processor
In-house Cheese
store
Contract Cheese
store
Processor
Packing plant
Processor
National
Distribution
Centre
Retailer
RDC
Retailer Stores
(550)
Retailer HQ
Consolidated
Demand
Ordering
Processor NDC
Customer
Services
Daily order -
SKU/Depot/
Vol
Sent @ 12.30-13:00
Delivery
orders
Processor HQ
Sales
Team/
Account
Manager
Processor HQ
Forecasting
Team
Processor HQ
Bulk Planning
Team
Cheese plant
Planner/Stock
office
Processor HQ
Milk
Purchasing
Team
Cheese plant
Transport
Manager
Actual
daily
delivery
figures
Daily
collection
planning
Weekly order for delivery to
Packing plant
Daily &
weekly Call-
off
Daily Call-off
15/day
22 pallet loads 15/day
A80
Shortages/
Allocation
instructions
Annual
Buying plan
Milk Availability
Forecast
Annual
prediction
of milk
production
Shortages/
Allocation
instructions
Daily milk
intake
Weekly milk
shortages
shortages
Spot mkt or
Processor
ingredients
Packing plant
Planning
Team
Processor HQ
JBA Invoicing
and
Sales Monitor
FGI and Last 5
weeks sales
Expedite
Changes
to existing
forecast -
exceptions
Retailer HQ
Retailer Buyer
Meeting
every 6
weeks
Packing plant
Cheese
ordering
10 day stock
plan
On line
stock info
7 day order
plan for bulk
cheese
Arrange
daily
delivery
schedule
Emergency
call-off
Daily
optimisation
of loads
Service
Monitor
Despatch and
delivery
confirmations
Processor NDC
Transport
Planning
Transport
Plan
Processor NDC
Inventory
MonitoringStock and
delivery
monitoring
Processor NDC
Warehouse
management
syatem
Operation
Instructions
Key
Shaded Boxes = Product flow system
Un-shaded boxes = Information flow system
RetailerCheese ProcessorFarms
Schedule
weekly &
Daily
10 Day
plan(wed) and
daily plan
15/day
Changes
to existing
forecast -
exceptions
Stock
availability
Monthly
review
Annual
f/cast
Source: IGD Food Chain Centre, February 2008
What about all the systems that already exist, built up over years?
A typical value chain diagram, showing the supply chain for cheese. Notice the extreme
complexity of information exchanges (in red)

6. Copyright Third Nature, Inc.
Even within one company it’s complicated
Off-the-shelf software, custom applications, SaaS
applications, logs, event streams, 3rd party datasets…

7. Copyright Third Nature, Inc.
Order Entry
Order
Database
Customer
Service
Integration
Program
Inventory
Database
Distribution
Integration
Program
Receivables
Database
Accounts
Receivable
Data
Warehouse
Analysts &
users
Data is already moving between many applications
but it isn’t streaming

8. Copyright Third Nature, Inc.
The shortcomings of the log-centered approach

9. Copyright Third Nature, Inc.
Supplement log-based models with non-intrusive collection

10. Copyright Third Nature, Inc.
Streaming infrastructure needs to enable reuse

11. Copyright Third Nature, Inc.
Image Attributions
Thanks to the people who supplied the images used in this presentation:
ignore_no_parking.jpg - http://www.flickr.com/photos/vermininc/2352068440
Streaming environment - http://www.noumenal.com/
phone booth red snow - http://www.flickr.com/photos/wwarby/5907617001/
lots of screens to watch – Cprbis stock photo
Slide 11

12. Copyright Third Nature, Inc.
About Third Nature
Third Nature is a research and consulting firm focused on new and emerging technology
and practices in analytics, business intelligence, and performance management. If your
question is related to data, analytics, information strategy and technology infrastructure
then you‘re at the right place.
Our goal is to help companies take advantage of information-driven management
practices and applications. We offer education, consulting and research services to
support business and IT organizations as well as technology vendors.
We fill the gap between what the industry analyst firms cover and what IT needs. We
specialize in product and technology analysis, so we look at emerging technologies and
markets, evaluating technology and hw it is applied rather than vendor market positions.