Microsoft StreamInsight, part of the recent SQL Server 2008 R2 release, is a new platform for building rich applications that can process high volumes of event stream data with near-zero latency. Mark Simms of Microsoft's SQLCAT will demonstrate the core skill sets and technologies needed to deliver StreamInsight enabled solutions, and discuss some of the core scenarios. Mark will provide a detailed walkthrough of the three major components of StreamInsight: input and output adapters, the StreamInsight engine runtime, and the semantics of the continuous standing queries hosted in the StreamInsight engine. This presentation includes hands-on demos, including building out a real-time data processing solution interacting with SQL Server and Sharepoint. You will learn: • The new capabilities StreamInsight brings to data processing and analytics, unlocking the ability to extract real time business intelligence from streaming data. • How StreamInsight interacts with and compliments other components of SQL Server and the rest of the Microsoft technology stack. • How to ramp up on the skills and technology necessary to build out end to end solutions leveraging streaming data sources.

1. SQL Server 2008 R2
StreamInsight
Speaker: Mark Simms
Microsoft SQLCAT
Silicon Valley SQL Server User Group
May, 2010
Mark Ginnebaugh, User Group Leader,
mark@designmind.com

2. masimms@microsoft.com

4. Load barrier is dictated by
current choices of the solution,
Facts/sec.
e.g., loading into databases,
persisting into files. This is
intrinsic because in current
approaches no processing can
be done till the data is loaded. Custom-built solutions that
carry huge development and 100000
customization costs
10000
Active DW analytics
1000
Traditional DW Analytics
100
years months days hrs min sec
Time of interest
Present
ET time in ETL Load time in ETL

5. Analytical results need to reflect important changes in
business reality immediately and enable responses to them
with minimal latency
Database Applications Event-driven Applications
Query Ad-hoc queries or Continuous standing
Paradigm requests queries
Latency Seconds, hours, days Milliseconds or less
Data Rate Hundreds of events/sec Tens of thousands of
events/sec or more
Query Declarative relational Declarative relational and
Semantics analytics temporal analytics
request Event
output
stream
input
stream
response
5

6. Latency
Months
StreamInsight
Days Target Scenarios
hours Relational Database Applications Operational Analytics
Applications, e.g., Logistics,
Minutes Data Warehousing etc.
Applications
Seconds
Web Analytics Applications
100 ms Monitoring Manufacturing
Financial trading
Applications Applications
< 1ms
Applications
0 10 100 1000 10000 100000 ~1million
Aggregate Data Rate (Events/sec.)
6

7. Manufacturing: Web Analytics: Financial Services: Power Utilities:
• Sensor on plant • Click-stream data • Stock & news feeds • Energy
floor • Online customer • Algorithmic trading consumption
• React through behavior • Patterns over time • Outages
device controllers • Page layout • Super-low latency • Smart grids
• Aggregated data • 100,000 events /sec • 100,000 events /sec • 100,000 events/sec
• 10,000 events/sec
Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds
Data Stream
Data Stream
Visual trend-line and KPI monitoring
Batch & product management
Automated anomaly detection
Real-time customer segmentation
Algorithmic trading
Proactive condition-based maintenance
Asset Specs & StreamInsight Engine
Parameters
• Threshold queries
Stream Data Store & • Event correlation from
Archive Lookup multiple sources
• Pattern queries
7

8. StreamInsight
Industry trends
advantage
• Data acquisition
costs are
Manage • Process data
Monitor KPIs incrementally,
negligible business via
Record raw i.e., while it is in
KPI-triggered
• Raw storage costs data (history) flight
are small and
actions
• Avoid loading
continue to while still doing
decrease the processing
you want
• Processing costs
are non-negligible Mine historical data • Seamless
querying for
• Data loading Devise new KPIs
monitoring,
costs continue to managing and
be significant mining
8

9. StreamInsight Application
Development
StreamInsight Application at Runtime
Event sources Event targets
Input Output
Devices, Sensors Adapters StreamInsight Engine Adapters Pagers &
Monitoring devices
Standing Queries KPI Dashboards,
SharePoint UI
Web servers
Query Query
Logic Logic
Trading stations
Event stores &
Databases Query
Logic
Stock ticker, news feeds Event stores & Databases

16. SELECT COUNT(*) FROM ParkingLot
WHERE type = ‘AUTO’
AND color = ‘RED’

17. red
cars
last hour
Doesn’t seem like a
great solution…

18. This is the streaming data paradigm in a nutshell –
ask questions about data in flight.

19. Engine
Adapters
Engine
Queries
Extensions
Host
visual debugger
API

20. expressed
question
data
data
question

21. Tell me the just the color of each car that passes.
var result = from car in carStream
select new
{
car.Color
};

22. Give me only trucks.
var result = from car in carStream
where car.Type == “Truck”
select car;

23. Tell me the number of cars passed
every 10 seconds.
var result = from win in carStream.TumblingWindow(
TimeSpan.FromSeconds(10))
select new
{
count = win.Count()
};

24. var result = from win in carStream.TumblingWindow(
TimeSpan.FromSeconds(10))
select new
{
count = win.Count()
};

26. Count the number of cars for each make
separately every 10 seconds.
var result = from car in carStream
group car by car.make into eachGroup
from win in carStream.TumblingWindow(
TimeSpan.FromSeconds(10))
select new
{
make = eachGroup.Key,
count = win.Count()
};

27. application time
Current Time Indicators

30. public void EnqueueEvent(SourceData d)
{
var ev = CreateInsertEvent();
ev.Payload = new MouseEvent { Id = d.id, Value = d.value };
ev.StartTime = d.timestamp;
Enqueue(ref ev);
}

31. public void EnqueueEvent(SourceData d)
{
if AdapterState
return
var ev = CreateInsertEvent();
ev.Payload = new MouseEvent { Id = d.id, Value = d.value };
ev.StartTime = d.timestamp;
Enqueue(ref ev);
}

32. public void EnqueueEvent(SourceData d)
{
if AdapterState
return
var ev = CreateInsertEvent();
if (ev == null) return;
ev.Payload = new MouseEvent { Id = d.id, Value = d.value };
ev.StartTime = d.timestamp;
Enqueue(ref ev);
}

33. public void EnqueueEvent(SourceData d)
{
if AdapterState
return
var ev = CreateInsertEvent();
if (ev == null) return;
ev.Payload = new MouseEvent { Id = d.id, Value = d.value };
ev.StartTime = d.timestamp;
if (Enqueue(ref ev) == EnqueueOperationResult.Full)
{
Ready();
return;
}
}

34. Use them wisely!

35. public class TimeWeightedAverage :
CepTimeSensitiveAggregate<double, double>
{
public override double
GenerateOutput(IEnumerable<IntervalEvent<double>> events,
WindowDescriptor windowDescriptor)
{
double avg = 0;
foreach (IntervalEvent<double> ev in events)
{
avg += intervalEvent.Payload *
(ev.EndTime - ev.StartTime).Ticks;
}
return = avg / (windowDescriptor.EndTime –
windowDescriptor.StartTime).Ticks;
}
}

37. To learn more or inquire about speaking opportunities, please contact:
Mark Ginnebaugh, User Group Leader
mark@designmind.com