0
SQL Server 2008 R2
  StreamInsight
  Speaker: Mark Simms
    Microsoft SQLCAT


 Silicon Valley SQL Server User Group
    ...
masimms@microsoft.com
Load barrier is dictated by
 current choices of the solution,
                                                            ...
Analytical results need to reflect important changes in
 business reality immediately and enable responses to them
 with m...
Latency
Months
                                                                 StreamInsight
Days                        ...
Manufacturing:         Web Analytics:          Financial Services:      Power Utilities:
                     • Sensor on ...
StreamInsight
Industry trends
                                                         advantage
• Data acquisition
  cost...
StreamInsight Application
 Development
                               StreamInsight Application at Runtime
 Event sources ...
SELECT COUNT(*) FROM ParkingLot
WHERE type = ‘AUTO’
AND color = ‘RED’
red
cars
             last hour




 Doesn’t seem like a
  great solution…
This is the streaming data paradigm in a nutshell –
          ask questions about data in flight.
Engine


Adapters
      Engine
Queries

  Extensions
Host

                        visual debugger
API
expressed

                   question
data
                   data
question
Tell me the just the color of each car that passes.




var result = from car in carStream
             select new
       ...
Give me only trucks.




var result = from car in carStream
             where car.Type == “Truck”
             select car;
Tell me the number of cars passed
                   every 10 seconds.




var result = from win in carStream.TumblingWind...
var result = from win in carStream.TumblingWindow(
                           TimeSpan.FromSeconds(10))
             selec...
Count the number of cars for each make
           separately every 10 seconds.


var result = from car in carStream
      ...
application time

Current Time Indicators
public void EnqueueEvent(SourceData d)
{
    var ev = CreateInsertEvent();

    ev.Payload = new MouseEvent { Id = d.id, V...
public void EnqueueEvent(SourceData d)
{
    if                  AdapterState


       return


    var ev = CreateInsertE...
public void EnqueueEvent(SourceData d)
{
    if                  AdapterState


       return


    var ev = CreateInsertE...
public void EnqueueEvent(SourceData d)
{
    if                  AdapterState


        return


    var ev = CreateInsert...
Use them wisely!
public class TimeWeightedAverage :
                      CepTimeSensitiveAggregate<double, double>
{
    public override d...
To learn more or inquire about speaking opportunities, please contact:

                Mark Ginnebaugh, User Group Leader...
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Microsoft StreamInsight
Upcoming SlideShare
Loading in...5
×

Microsoft StreamInsight

2,928

Published on

Microsoft StreamInsight, part of the recent SQL Server 2008 R2 release, is a new platform for building rich applications that can process high volumes of event stream data with near-zero latency.

Mark Simms of Microsoft's SQLCAT will demonstrate the core skill sets and technologies needed to deliver StreamInsight enabled solutions, and discuss some of the core scenarios.

Mark will provide a detailed walkthrough of the three major components of StreamInsight: input and output adapters, the StreamInsight engine runtime, and the semantics of the continuous standing queries hosted in the StreamInsight engine.

This presentation includes hands-on demos, including building out a real-time data processing solution interacting with SQL Server and Sharepoint.

You will learn:

• The new capabilities StreamInsight brings to data processing and analytics, unlocking the ability to extract real time business intelligence from streaming data.

• How StreamInsight interacts with and compliments other components of SQL Server and the rest of the Microsoft technology stack.

• How to ramp up on the skills and technology necessary to build out end to end solutions leveraging streaming data sources.

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,928
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
129
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "Microsoft StreamInsight "

  1. 1. SQL Server 2008 R2 StreamInsight Speaker: Mark Simms Microsoft SQLCAT Silicon Valley SQL Server User Group May, 2010 Mark Ginnebaugh, User Group Leader, mark@designmind.com
  2. 2. masimms@microsoft.com
  3. 3. Load barrier is dictated by current choices of the solution, Facts/sec. e.g., loading into databases, persisting into files. This is intrinsic because in current approaches no processing can be done till the data is loaded. Custom-built solutions that carry huge development and 100000 customization costs 10000 Active DW analytics 1000 Traditional DW Analytics 100 years months days hrs min sec Time of interest Present ET time in ETL Load time in ETL
  4. 4. Analytical results need to reflect important changes in business reality immediately and enable responses to them with minimal latency Database Applications Event-driven Applications Query Ad-hoc queries or Continuous standing Paradigm requests queries Latency Seconds, hours, days Milliseconds or less Data Rate Hundreds of events/sec Tens of thousands of events/sec or more Query Declarative relational Declarative relational and Semantics analytics temporal analytics request Event output stream input stream response 5
  5. 5. Latency Months StreamInsight Days Target Scenarios hours Relational Database Applications Operational Analytics Applications, e.g., Logistics, Minutes Data Warehousing etc. Applications Seconds Web Analytics Applications 100 ms Monitoring Manufacturing Financial trading Applications Applications < 1ms Applications 0 10 100 1000 10000 100000 ~1million Aggregate Data Rate (Events/sec.) 6
  6. 6. Manufacturing: Web Analytics: Financial Services: Power Utilities: • Sensor on plant • Click-stream data • Stock & news feeds • Energy floor • Online customer • Algorithmic trading consumption • React through behavior • Patterns over time • Outages device controllers • Page layout • Super-low latency • Smart grids • Aggregated data • 100,000 events /sec • 100,000 events /sec • 100,000 events/sec • 10,000 events/sec Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Data Stream Data Stream Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Asset Specs & StreamInsight Engine Parameters • Threshold queries Stream Data Store & • Event correlation from Archive Lookup multiple sources • Pattern queries 7
  7. 7. StreamInsight Industry trends advantage • Data acquisition costs are Manage • Process data Monitor KPIs incrementally, negligible business via Record raw i.e., while it is in KPI-triggered • Raw storage costs data (history) flight are small and actions • Avoid loading continue to while still doing decrease the processing you want • Processing costs are non-negligible Mine historical data • Seamless querying for • Data loading Devise new KPIs monitoring, costs continue to managing and be significant mining 8
  8. 8. StreamInsight Application Development StreamInsight Application at Runtime Event sources Event targets Input Output Devices, Sensors Adapters StreamInsight Engine Adapters Pagers & Monitoring devices Standing Queries KPI Dashboards, SharePoint UI Web servers Query Query Logic Logic Trading stations Event stores & Databases Query Logic Stock ticker, news feeds Event stores & Databases
  9. 9. SELECT COUNT(*) FROM ParkingLot WHERE type = ‘AUTO’ AND color = ‘RED’
  10. 10. red cars last hour Doesn’t seem like a great solution…
  11. 11. This is the streaming data paradigm in a nutshell – ask questions about data in flight.
  12. 12. Engine Adapters Engine Queries Extensions Host visual debugger API
  13. 13. expressed question data data question
  14. 14. Tell me the just the color of each car that passes. var result = from car in carStream select new { car.Color };
  15. 15. Give me only trucks. var result = from car in carStream where car.Type == “Truck” select car;
  16. 16. Tell me the number of cars passed every 10 seconds. var result = from win in carStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { count = win.Count() };
  17. 17. var result = from win in carStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { count = win.Count() };
  18. 18. Count the number of cars for each make separately every 10 seconds. var result = from car in carStream group car by car.make into eachGroup from win in carStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { make = eachGroup.Key, count = win.Count() };
  19. 19. application time Current Time Indicators
  20. 20. public void EnqueueEvent(SourceData d) { var ev = CreateInsertEvent(); ev.Payload = new MouseEvent { Id = d.id, Value = d.value }; ev.StartTime = d.timestamp; Enqueue(ref ev); }
  21. 21. public void EnqueueEvent(SourceData d) { if AdapterState return var ev = CreateInsertEvent(); ev.Payload = new MouseEvent { Id = d.id, Value = d.value }; ev.StartTime = d.timestamp; Enqueue(ref ev); }
  22. 22. public void EnqueueEvent(SourceData d) { if AdapterState return var ev = CreateInsertEvent(); if (ev == null) return; ev.Payload = new MouseEvent { Id = d.id, Value = d.value }; ev.StartTime = d.timestamp; Enqueue(ref ev); }
  23. 23. public void EnqueueEvent(SourceData d) { if AdapterState return var ev = CreateInsertEvent(); if (ev == null) return; ev.Payload = new MouseEvent { Id = d.id, Value = d.value }; ev.StartTime = d.timestamp; if (Enqueue(ref ev) == EnqueueOperationResult.Full) { Ready(); return; } }
  24. 24. Use them wisely!
  25. 25. public class TimeWeightedAverage : CepTimeSensitiveAggregate<double, double> { public override double GenerateOutput(IEnumerable<IntervalEvent<double>> events, WindowDescriptor windowDescriptor) { double avg = 0; foreach (IntervalEvent<double> ev in events) { avg += intervalEvent.Payload * (ev.EndTime - ev.StartTime).Ticks; } return = avg / (windowDescriptor.EndTime – windowDescriptor.StartTime).Ticks; } }
  26. 26. To learn more or inquire about speaking opportunities, please contact: Mark Ginnebaugh, User Group Leader mark@designmind.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×