This document discusses big fast data and perishable insights that must be acted on quickly. It describes streaming analytics as analyzing data in motion from real-time sources to gain opportunities and avoid crises. Examples of use cases provided are real-time market surveillance and an IoT-enabled smart bank that can send targeted offers to customers. Key components discussed are storing high velocity data in HBase and analyzing it using Storm or Spark on Kafka.

1. Big Fast Data
Enabling Perishable Insight at Scale

2. Perishable Insights
1.Data is perishable if there is only a limited
amount of time to act upon that event.
Market data, clickstream, mobile devices,
sensors, and transactions may contain valuable,
but perishable insights.
1.Sieze opportunities and avoid crises amidst
explosive complexity.

3. Streaming Analytics
“Streaming analytics is anything but a sleepy, rearview mirror analysis of
data. No, it is about knowing and acting on what’s happening in your
business at this very moment — now. Forrester calls these perishable
insights because they occur at a moment’s notice and you must act on
them fast within a narrow window of opportunity before they quickly lose
their value. The high velocity, white-water flow of data from innumerable
real-time data sources such as market data, Internet of Things, mobile,
sensors, clickstream, and even transactions remain largely unnavigated
by most firms. The opportunity to leverage streaming analytics has never
been greater.”

4. Big Fast Data
Big Data
Enable analytics of data at scale
PB's of data
Batch Processing (minutes to hours)
High latency
Fast Data
Enable analytics of data in motion
Millions of events per second
Stream Processing (nanoseconds to seconds)
Low latency
Data
Explosion
Computational
Explosion

5. Big Fast Data
lBatch Processing
lIn a traditional query model, you store data and then
run queries on the data as needed.
lQuery-driven model
lStream Processing
lIn a streaming data model, you store queries and then
continuously run data through the queries.
lEvent-driven model

6. Thinking in Streams
lFiltering
lStreaming data can be filled with irrelevant or invalid
data since data typically does not go through a data
governance step until it is stored in Hadoop.
lAggregation/Correlation
lStreaming data typically comes from multiple sources
and can be combined in multiple ways.
lLocation/Motion
lMobile devices, from phones to wearables, are
ubiquitous.

7. Thinking in Streams
lTime Windows
lBy taking a snapshot of the data stream, time
windows can be used to provide time series analysis
in real time (weighted moving averages, Bollinger
Bands, etc)
lTemporal Patterns
lEvents can often contain interesting patterns relative
to new data coming in, such as data streaming at
different times and different patterns of data at the
same time.

8. Use Case for Fast Data Analytics
lManage Risk
lMarket Surveillance in a high frequency world
lMaximize Reward
lCustomer Satisfaction in the Internet of Things

9. Real-Time Market Surveillance
lConvergent threat system
lSupport for historical, realtime and predictive
modeling
lSupport for Big Fast Data
lSupport for multi- and cross-asset class
monitoring
lSupport for cross-border surveillance
l

10. Real-Time Market Surveillance
lContinuous predictive analysis
lBy leveraging historical data, a continuous
predictive analysis can extrapolate events that
have happened up to the current time and then
predict a future event.
lFor example, the normal operating parameters of
an algorithm (size, frequency, instrument, order-
to-trade ratio) can be established strictly based on
history. Deviation from this norm can trigger
defensive measures. Consider Knight Capital.

11. Real-Time Market Surveillance
lConvergent threat system
lA rogue algorithm can be shut down if its operating
outside of its normal behavior pattern.
lA rogue trader, market manipulator or insider dealer can
be identified based on rules specified by Compliance.
lThe key is a unified framework.
lProvide a unified data ingestion system to ingest data
from the large number of disconnected (and
unconnectable) data sources.

12. Disintermediation
lMobile wallets pose the same threat level to
banks as self-directed equities trading.
lDBS uses its vast amount of client infromation to
categorize customers, statistically predict behavior
and send targeted promotional offers to their
client's cell phones when they are engaged with a
business partner.

13. IoT Smart Bank
lA location-aware promotion application requires
continuous, event-driven queries that continuously
monitoring , analyzing and responding to the
changing status of subscribers and promotions.
lThese queries are multidimensional; matching
location, context, preferences, socioeconomic
factors, and spending habits. These dimensions
are in flux.

14. IoT Smart Bank
lThe numbers
l1,000 people moving once per minute equals 17
events per second
l1 million people would generate 16,667 events
per second
lWells Fargo has 70 million customers.

15. Streaming Use Cases
lNetwork monitoring
lIntelligence and surveillance
lRisk management
lE-commerce
lFraud detection
lSmart order routing
lTransaction cost analysis
lPricing and analytics
lMarket data management
lAlgorithmic trading
lData warehouse augmentation

16. Components of Fast Data Analytics
lTo store data at velocity, Flume into Hbase
lTo analyze data at velocity, Kafka in Storm or
Spark

17. Open Source Landscape
lApache Storm
lProvides massively scalable event collection
lApache Spark
lGeneral framework for large-scale data processing
lApache Samza

18. Apache Storm

19. Apache Spark

20. Commercial Landscape
lCommercial products will differ from open source
products in the following areas:
lDevelopment tools
lBusiness applications and platform integration
lImplementation support
lCompany financials
lLicensing

21. Commercial Landscape
lIBM InforSphere Streams
lInformatica Platform for Streaming Analytics
lSAP Event Stream Processor
lSoftware AG IBO
lSQLstream Blaze
lTibco Streambase
lVitria Operational Intelligence