SQL Server 2008 R2 StreamInsight

Petabytes for Peanuts! Making sense of “Ambient Data” SQL Server Stream Insight
Ing. Eduardo Castro, PhD
Comunidad Windows
ecastro@grupoasesor.net
http://ecastrom.blogspot.com

Key Takeaways…
Massive shift in how we process data
Incredible data volumes
Remaking how we discover
Changing the Scientific Method
Reducing latency & impedance
Extreme Scale Data Processing
Stream Processing (Several Views)
From “programs” to “queries”
What’s up with this “anti-SQL” stuff anyhow?

1997
Storage Cost: $~1.00
Transfer Time: ½ hour
2009
Storage Cost: ~0.1₵
Transfer Time: 8 sec.
1982
Storage Cost: $~2000
Transfer Time: 1 day
“Free” Storage Power

Ambient Data?
Over 84 percent of Americans have cell phones, according to Steve Largent, president and CEO of CTIA. While two trillion minutes were used in 2007, an 18 percent increase over 2006 talk times.
More than 48 billion text messages were sent in the month of December 2007, an average 1.6 billion messages per day. The rate of text messaging represented a 157 percent increase over December 2006 texting.
http://www.clickz.com/3628985
Text Message Traffic in US: 160GB / day  58TB / year
Voice traffic in US (GSM encoding)
200PB / year

The Old World
Data volumes constrained by human typing speed
App & Data formed closed system
App
Assume 200M people in US typing 8 hr / day @ 10K keystokes / hour:
2TB/hror ~6PB / year
DB

The Old New World
Available data exploded
Available Data
Questions toAnswer
What data shouldwe throw out?
Design Schema
Design ETL
What if we have
a new question?
DW Nirvana!

The New World of Abundant Data
Save All Available Data
Hypothesize  Theorize  Test
New Question to Answer
AlgorithmicProcessing
Run “query”
over data…
Exploit
Correlation…
Correlation is
Enough!
Analyze reduced data
The CMS front end of the Large Hadron Collider records 1TB/sec!
http://blogs.discovermagazine.com/cosmicvariance/2006/09/27/lhc-factoids/
Interesting Read: The Petabyte Age: Because More Isn't Just More — More Is Different
http://www.wired.com/science/discoveries/magazine/16-07/pb_intro

Analyze  Model  Monitor
1
Event Stream both stored and processed
Event Processing
Engine
4
Produce real time alerts and action
Event Stream
Alerts & Action
3
Models installed in event processing engine
Correlation Model
2
Analysis produces event correlation models
Analysis

Extreme Scale Data Processing
Source
DW
Traditional Data Warehouse
Source
Source
ETL
Source
Source
Analysis / Reporting
Source
Source
Extreme ScaleData Processing
DW
Non-traditional
Sources
1
2
Majority of data filtered or discarded
All data retained and reprocessed
Analysis / Reporting
Analysis

SQL Server 2008 R2 – StreamInsight Technology
Data volumes are exploding with event data streaming from sources such as RFID, sensors and web logs
The size and frequency of the data make it challenging to store for data mining and analysis.
The ability to monitor, analyze and take business decisions in near real-time

SQL Server StreamInsight’s
SQL Server StreamInsight’s ability to derive insights from data streams and act in near real time provides significant business benefits. Some of the possible scenarios include:
Algorithmic trading and fraud detection for financial services
Industrial process control (chemicals, oil and gas) for manufacturing
Electric grid monitoring and advanced metering for utilities
Click stream web analytics
Network and data center system monitoring.

.NET
C#
LINQ
StreamInsight Application Development
StreamInsight Application at Runtime
Event sources
Event targets
Input
Adapters
Output
Adapters
StreamInsight Engine
Devices, Sensors
Pagers &
Monitoring devices
Standing Queries
KPI Dashboards, SharePoint UI
Web servers
Query Logic
Query Logic
Trading stations
Event stores & Databases
Query Logic
Event stores & Databases
Stock ticker, news feeds
StreamInsight Platform

Events
Represent the user payload along with temporal characteristics
Streams
Sequence of events
Flows into (one or more) standing queries in StreamInsightengine
Queries
Operate on event streams
Apply desired semantics on events
Adapters
Convert custom data from event sources to / from StreamInsight events
Key Concepts

Event
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
request
output stream
input stream
response
What is CEP?

Latency
Relational Database Applications
CEP Target Scenarios
Operational Analytics Applications, Logistics, etc.
Data Warehousing Applications
Web Analytics Applications
Manufacturing Applications
Financial Trading Applications
Monitoring Applications
Aggregate Data Rate (Events/sec)
Event Processing Scenarios

Use Case: Customer Segmentation
Analysis of Click Streams on MSN.com
Web Server log streamed into StreamInsight
Categorizing user behavior based on URL:
Click targets
Search keywords
Segmentation of user IDs into markets
Adapting navigational structure and ad placement in real time
Patterns over time windows: user first clicks PageA, then PageB, then PageC within X seconds
High performance requirements
Millions of online users
Low latency (seconds)
Possible late events

Use Case: NBC Sunday Night Football
1
Telemetry Receiver
4
StreamInsight
Listener Adapter
GeoTag and group by region
SQL Adapter
PerfCounter Adapter
2
Count total events
Count session starts
Count active sessions
3

Use Case: Data Center
Power Consumption
Visualize
Process Information
Complex Aggregations/
Correlations
Central
time series
archive
Query
ETW
Input Adapter
Query
2
1
Query
Power Meter
Input Adapter
3

ChallengesHow do I …
detect interesting patterns?
reason about temporal semantics?
correlate data?
aggregate data?
avoid writing custom imperative code?
create a runtime environment for continuous and event-driven processing?
As a developer, I need a platform!

Query Expressiveness
Selection of events (filter)
Calculations on the payload (project)
Correlation of streams (join)
Stream partitioning (group and apply)
Aggregation (sum, count, …) over event windows
Ranking over event windows (topK)

Projection
Filter
Correlation (Join)
Aggregation over windows
Group and Aggregate
Query Expressiveness
var result = from e ininputStream
group e by e.id intoeachGroup
from win ineachGroup.TumblingWindow(
TimeSpan.FromSeconds(10))
selectnew { eachGroup.Key,
avg = win.Avg(e => e.W) };

Conclusion
CEP Platform & API
Event-triggered, fast Computation
API for Adapters, Queries, Applications
Declarative LINQ
Flexible Adapter API
Extensible
Supportability

Links
http://comunidadwindows.org
http://ecastrom.blogspot.com
http://www.microsoft.com/sql

SQL Server 2008 R2 StreamInsight

by Eduardo Castro on May 21, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 23

Statistics

SQL Server 2008 R2 StreamInsight — Presentation Transcript