Your SlideShare is downloading. ×
  • Like
Microsoft SQL Server - StreamInsight Overview Presentation
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Microsoft SQL Server - StreamInsight Overview Presentation



Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Now more than ever, organizations have the need to use information management to compete and grow in a difficult market by reducing costs and identifying the highest value opportunities for their organization.  As we engage with customers, you explain the pressures you have to deliver more real-time information through rich applications while also reducing costs in this new economy. The converging IT trends of virtually free storage space, the rapid adoption of virtualization and the increasing capabilities of industry standard hardware, the emergence of the Cloud as a deployment option, and the need for real-time business information for all employees through easy to use tools are driving the data explosion we see today. You need a complete approach to managing, accessing and delivering information across your organization to accelerate and improve business decisions..  We also understand that at the center of delivering on these business needs there are people - IT Professionals, who support the expanding information needs of the business through IT services, Developers who build solutions in time to capture business opportunities in an increasingly competitive market and Business Users, who quickly mine volumes of data for business insights to help increase customer satisfaction and drive business results. Microsoft, along with our worldwide partners, is committed to deliver an Information Platform which enables your people and provides you with a complete set of technologies and tools to help you realize more value from your information at the lowest total cost of ownership. As we invest in the Information Platform we think about what we need to deliver to you in each of these 4 areas:  Mission-Critical Platform Mission-Critical Platform for all your application requirements, delivered with the industry’s best TCOEnsuring your applications and systems are reliable, highly available, secure and deliver superior, predictable performance is top priority for IT and for Microsoft. At our core, we are focused on delivering an information platform which meets the needs for performance, scalability, availability, and security for your most mission critical business applications at a lower cost of ownership, including acquisition costs and the ongoing costs of support, management and maintenance. From the server to the datacenter we offer our customers a reliable information platform that grows with their needs at a third of the cost of our enterprise DB competitor.Secure and available infrastructureWe have seen tremendous growth in our Enterprise business – investing significantly in our datacenter capability on the Windows Server platform, building out the industry’s most widely deployed database platform. The largest enterprises on the planet are now running their top-tier applications on the Microsoft platform and seeing sustained high-performance and uptime that directly benefits their businesses’ bottom line. Scales to all business requirementsWith storage costs declining, and increasing data volumes, IT departments need the ability to scale up seamlessly to process an increasing number of transactions, as well as storing and processing larger volumes of information. Microsoft’s Information Platform provides high scalability and performance for OLTP and BI as well as new Data Warehousing capabilities to scale to hundreds of terabytes to deliver on the most critical business needs. A complete and interoperable platform that empowers IT to be more productive and agile The boundaries of the IT environment continue to be stretched and IT Professionals have increased responsibility to manage application requirements for the datacenter, across mobile devices and the desktop, and now out in the cloud. To successfully manage these resources across the enterprise IT professionals require a consistent, and productive platform, including easy to use and robust tools.  Increased DBA Productivity and ControlSQL Server has long had a focus on DBA productivity – by providing rich management tools that empower administrators to take control of their environment. We continue to invest in tools that enable administrators to manage multiple servers at once via policies and to proactively manage system performance. We’re also investing to provide complementary tools for areas such as data integration and master data management.  Consolidation and VirtualizationConsolidation and virtualization can deliver significant resource and operational benefits in the datacenter, particularly at a time when IT professionals are facing budget pressure along with the need to continue the same level of service. Together, Windows Server, System Center and SQL Server can deliver a virtualized Data Center, providing increased utilization of server resources to reduce costs, greater standardization to reduce administration overhead and greater agility to dynamically scale to support changing and growing business needs.  The Power of Choice - Harnessing the CloudThe cloud offers new ways for IT to reduce the friction of connecting with customers and partners, the ability to quickly provision resources, and a way to reduce operational costs. Microsoft is delivering SQL Azure as part of the Windows Azure Platform to enable IT to manage these resources with a platform and tools that are consistent with their existing SQL Server environment.Dynamic DevelopmentIntegrated tools to help Developers more quickly build rich, intuitive and connected applications Developers are continually asked to deliver richer applications and services while lowering the time-to-solution to adapt to business needs and opportunities. With Visual Studio, the .NET Framework, and SQL Server developers have a highly productive platform to deliver data through their applications and to collaborate more efficiently with IT on deployment needs and requirements. The consistent development experience also gives developers the power to build applications for deployment across devices, desktops, the datacenter, and the cloud.  Tight Integration across the stackSQL Server has deep integration with Visual Studio and .NET to provide developers with a rich, seamless development experience. Developers can use familiar tools and the power of a model-driven approach to vastly improve their productivity, by abstracting away from database specifics and instead focusing on the business logic and outcomes required.  Platform for all DataBeing able to bring unstructured data alongside structured data in a single application gives developers the power to bring the relational model to new types of data. They can extend existing applications and develop new types of applications that incorporate location aware services, real time streaming data – all in a secure, synchronized way.  Rich and embeddable user experiencesDevelopers can embed powerful user experiences into their applications through components such as spatial data for rich mapping scenarios, embeddable BI controls and data synchronization. Looking ahead, developers can integrate value-added cloud services into applications for a differentiated value prop and additional services opportunities.Pervasive InsightA complete Business Intelligence platform that enables users across the organization to derive greater insights through familiar toolsBusiness Intelligence continues to be the #1 priority for CIOs. Even as budgets are cut IT is being asked to do more to deliver information to the business. Getting the right information to the right person at the right time is critical for success now more than ever. This requires providing access through familiar tools to end users while also providing IT the ability to manage and secure access to the information. By enabling access to all types of users through familiar tools, in a secure and well managed way – businesses can get more value from their information than ever before. Microsoft helps companies consolidate information, deliver access and empower users at all levels of the business: strategic, tactical and operational decision makers. Integrate all your data and scale your reporting systemsWith SQL Server, you can integrate the many sources of data in your organization into large scale data warehouses and deliver rich, high-performance reporting to your whole organization.  Enable users to derive insights through familiar toolsIncreased processing power and memory for desktops and laptops are enabling information workers to do more advanced data mining and analysis to improve decision making. We are extending our capabilities to deliver the power of self-service business intelligence to individual users through the familiar tools they use everyday such as Microsoft Excel. Enable collaborative decision making Today organizations have relevant data that lives in the datacenter, on user devices and desktops, and through external services. It is critical for users to be able to draw upon the wealth of information that is distributed throughout the organization and to do analysis and reporting with the data sources that are most relevant. The real value is that individual insight can be easily shared across work groups and across the organization. With Microsoft Office SharePoint, individual information workers can share and publish insights and analysis tools to scale reach and deliver greater value to the organization.ClosingLooking across the Information Platform, we think about how these capabilities can better support our customers’ information needs. For example a national retailer could use technology to track the demographics, volume and location of customers throughout stores across the country.  By processing large volume of data in real-time, they could then evaluate the needs of their customer base at any given moment and make changes to promotions or the store environment like the background music, to better appeal to the audience and drive real-time purchase.  Looking across the Information Platform, the ability to dynamically respond to customer needs could be built by working with your development organization to develop a new application or extending an existing application, partnering with IT to support the application through greater utilization of existing assets, and delivering information to business users through familiar tools. By providing business information pervasively throughout the organization in real-time this organization would be able to much more quickly identify and capture business opportunities.The Information Platform delivers the rich capabilities of SQL Server and SQL Azure to enable IT, developers and end users to deliver on your most critical business needs and maximize the value of your information throughout the organization at a low cost of ownership.
  • The purpose of this slide is to communicate the next wave of SQL innovation and how it should be messaged. There are three key areas of innovation: SQL Server 2008 R2 – the next generation release of the SQL Server database platform, Parallel Data Warehouse, appliances for high scale data warehousing and SQL Azure – scalable relational database platform service in the cloud. When you talk to each of these innovations, you want to emphasize that each is built on the solid foundation of SQL Server 2008.SQL Server 2008 R2 enables businesses to make more timely, better informed decisions by enabling users to access, integrate, analyze and share information using tools they are already familiar with – Microsoft Office. DBAs can more efficiently manage databases and applications at scale with multi-server management. Enhanced support for Virtualization will provide opportunities to further optimize resources through consolidation.SQL Azure – delivers on the Microsoft Data Platform vision of extending the SQL Server capabilities to the cloud as web-based services. SQL Azure provides an internet-facing database and advanced query processing services and is ideal solution for customers building new applications or integrating with existing investments in the cloud. Built on SQL Server technologies, SQL Azure provides the same enterprise-class availability, scalability , and security with the benefits of built-in data protection, self healing and disaster recovery as well as a consistent SQL Server programming model & tools. SQL Azure is scheduled for release later this year.Parallel Data Warehouse is a highly scalable data warehouse appliance. It uses Massively Parallel Processing (MPP) to deliver high performance and scalability on SQL Server 2008, Windows Server 2008 and industry-standard hardware. Parallel Data Warehouse is a separate release/edition but will launch jointly with R2 in the first half of 2010.
  • The “hero” offering that we’ll really sell broadly is SQL Server 2008 R2. You’ll notice that the messaging pillars for R2 are aligned with the Trusted, Productive, Intelligent messaging pillars for SQL Server 2008.To provide a comprehensive view of the R2 release (especially for SA discussions) this slide focuses on the key technology investments for each of the pillars. Trusted, Scalable Platform, it’s important to re-emphasize the enterprise-level security, high availability and scale delivered in SQL 2008 for mission critical applications. What’s new to the R2 release is StreamInsight for Complex Event Processing, Master Data Services for data consistency across heterogeneous systems, SQL Server support for up to 256 Logical Processors – providing greater flexibility in hardware choice, and enhanced data compression including Unicode compression for greater storage efficiency.IT & Developer Efficiency, the release-defining scenarios for R2 are Virtualization and Multi-server Management to increase IT productivity and reduce data management costs. This is enabled by the new manageability enhancements delivered in Management Studio plus R2 support for Windows Server 2008 R2 Hyper-V Virtualization and Live Migration. With R2, we introduce a new concept to the market, a single unit of deployment which will significantly increase the efficiencies around data-tier application deployments and upgrades resulting in dramatically streamlined consolidation and management efforts. Managed Self Service BI – Self-service analytics is delivered with PowerPivot – which is about enabling users to get the data they need without IT having to do a 6-month project to create a new application. PowerPivot gives users the power and flexibility to take internal data, integrate it with other information – and within Excel do all the slicing and dicing you need to get to the view of information you really need for decision making. This is all done “in memory” through really high compression in RAM – which is very powerful. None of the other major BI vendors can do this. User can easily share their application with others by publishing it to SharePoint.Once published, there’s a PowerPivot Operations Dashboard in SharePoint that provides IT Pros with the ability to manage business intelligence through SharePoint like it was just another service they provide. This new dashboard allows IT Pros to look at which reports are being run, who’s running them, how often are they running them, is the server being over utilized, how can I move the app dynamically to another server…BI is basically another managed object for IT. The reports the users publish is the object and we enable IT to understand how it is being used and apply rules and policies to that.
  • Your free PASS membership enables access to 12 Virtual Chapters, 24 Hours of PASS broadcasts, local PASS Chapters, professional development resources, newsletters, events calendar, and much moreMeet up with like-minded professionals throughout the year at face-to-face PASS Chapter meetings, discuss topical SQL Server issues, share tips and tricks, network, enjoy special guest speakers and get access to all things PASSAt PASS Summit,meet top industry experts , learn about best practices, effective troubleshooting, how to prevent issues, save money, and build a better SQL Server environment


  • 1. SQL Server 2008 R2 StreamInsight Overview
    Microsoft Corporation
  • 2. Presentation Outline (hidden slide)
    • Technical Level: 200
    • 3. Intended Audience: ITDMs,DBAs, Database Developers, Database Architects, BI specialists
    • 4. Objectives (what do you want the audience to take away):
    • 5. Introduction to Complex Event Processing (CEP) and Microsoft’s StreamInsight solution
    • 6. Technical overview of CEP architecture and development environment
    • 7. Presentation Outline:
    • 8. Overview - Information Platform Vision, SQL Server 2008 Foundation, SQL Server 2008 R2 Value Proposition
    • 9. Introduction to Complex Event Processing (CEP)
    • 10. Business Challenges and key CEP benefits
    • 11. Platform Architecture
    • 12. CEP and LINQ Queries
    • 13. CEP Development Environment
    • 14. Query Binding & Configuration
    • 15. CEP Platform Summary
    • 16. Deployment Scenarios
    • 17. Roadmap
  • Information Platform Vision
    Pervasive Insight
    Empowered IT
    Dynamic Development
    Desktop & Mobile
    Server & Datacenter
  • 18. Scalable relational database platform
    Consistent, familiar model & tools
    Self-managed, highly available cloud services
    MPP support for 10s to 100s TB DW
    Highly scalable appliances
    Seamless integration with Microsoft BI
    A Comprehensive Platform
    Managed Self-Service BI
    Multi-server management
    Virtualization & Live Migration
    Enterprise-level security, scalability
    High-scale, complex event processing
    Data consistency across heterogeneous systems
    Multi-server management
    Virtualization & Live Migration
    Accelerated development& deployment
    Self-service analytics
    Self-service reporting
    Streamlined collaboration& management
    MPP support for 10s-100s TB DW
    Highly scalable appliances
    Seamless BI Integration
    Scalable relational database platform
    Consistent, familiar model & toolsSelf-managed, highly available cloud service
  • 20. What is CEP?
    Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
    output stream
    input stream
  • 21. Latency
    Scenarios for Event Processing
    Relational Database Applications
    CEP Target Scenarios
    Operational Analytics Applications, e.g., Logistics, etc.
    Data Warehousing Applications
    Web Analytics Applications
    Manufacturing Applications
    Financial trading Applications
    Monitoring Applications
    Aggregate Data Rate (Events/sec.)
  • 22. Example CEP Scenarios
    Power, Utilities:
    • Sensor on plant floor
    • 26. React through device controllers
    • 27. Aggregated data
    • 28. 10,000 events/sec
    Web Analytics:
    • Click-stream data
    • 29. Online customer behavior
    • 30. Page layout
    • 31. 100,000 events /sec
    Financial Services:
    • Stock & news feeds
    • 32. Algorithmic trading
    • 33. Patterns over time
    • 34. Super-low latency
    • 35. 100,000 events /sec
    Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds
    Data Stream
    Data Stream
    Visual trend-line and KPI monitoring
    Batch & product management
    Automated anomaly detection
    Real-time customer segmentation
    Algorithmic trading
    Proactive condition-based maintenance
    Stream Data Store & Archive
    Event Processing Engine
    Asset Specs & Parameters
    • Threshold queries
    • 36. Event correlation from multiple sources
    • 37. Pattern queries
  • 38. Virtuous Cycle: Monitor, Manage, Mine
  • 39. Microsoft’s CEP Solution
    Data Sources, Operations, Assets, Feeds, Sensors, Devices
    Input Data Streams
    OutputData Streams
    Input Data Streams
    CEP Engine
    & Record
    & Design
    Operational Data Store & Archive
    CEP Engine
  • 40. CEP Deployment Alternatives
    CEP for lightweight processing and filtering
    CEP for aggregation and correlation of in-flight events
    CEP for complex analytics including historical data
    Event processing engines are deployed at multiple places on different scales
    • At the edge – close to the data source
    • 41. In the mid-tier – consolidate related data sources,
    • 42. In the data center – historical archive, mining, large scale correlation
    Web servers
    Complex Analytics &
  • 43. Platform from Microsoft for continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with low latency
    Pagers & Monitoring devices
    Devices, Sensors
    MSFT CEP Engine
    Input Adapters
    Output Adapters
    Web servers
    KPI Dashboards, SharePoint UI
    Event stores & Databases
    Trading stations
    Overview: CEP Platform from Microsoft
    Reference data
    Stock tickers & News feeds
    Event stores & Databases
    • Event driven
    • 44. Low latency processing (< milliseconds)
    • 45. High volume (>100,000 events /second)
    • 46. Correlated events from multiple sources
    • 47. Time interval oriented standing queries
    • 48. Manufacturing: Process monitoring i.e. scheduling, control, inventory
    • 49. Financial: algorithmic trading, compliance monitoring, fraud detection
    • 50. Web: Click stream analysis ,ad-delivery model
    • 51. Utility: Grid monitoring
    • 52. Healthcare: Patient Monitoring
    • 53. Networking: traffic monitoring ,security intrusion detection
    • 54. Telecom: Network monitoring
    • 55. Oil and Gas: Geological data analysis, process control
  • Events
    Events expose different temporal characteristics
    Point in time events
    Interval events with fixed duration
    Interval events with initially unknown duration
    Rich payloads capture all properties of an event
    Payload/ value 
    Time 
  • 56. Event Types
    Events in Microsoft’s CEP platform use the .NET type system
    Events are structured and can have multiple fields
    Fields are typed using the .NET framework types
    CEP engine provisioned timestamp fields capture all the different temporal event characteristics
    Event sources populate time stamp fields
  • 57. Event Streams
    A stream is a possibly infinite sequence of events
    Insertions of new events
    Changes to event durations
    Examples: time series sensor data, log entries
    Stream characteristics:
    Event/data arrival patterns
    • Steady rate with end-of-stream indication Examples: files, tables
    • 58. Intermittent, random, or in burstsExample: retail scanners, web, weather telemetry
    Out of order events
    • Order of arrival of events does not match the order of theirapplication timestamps
    Takeaway: CEP engine does the heavy lifting for you when dealing with out-of-order data
  • 59. Event Stream Adapters
    Design time specification of an adapter
    • for a particular event type and particular source/sink
    • 60. methods to handle event and stream behavior, and
    • 61. properties to indicate adapter features to the engine
    Adapter features
  • 62. Core CEP Query Engine
    CEP Engine
    Hosts “standing queries”
    Operators consume and produce streams
    Queries are composable
    Query results are computed incrementally
    Standing Queries
    Output Adapters
    Input Adapters
    Query instance management:
    Submit, start, stop
    Runtime statistics
    Takeaway: CEP engine does the heavy lifting for you when processing temporal event data
  • 63. Typical CEP Queries
    Typical CEP queries require combination of functionality
    Complex type describes event properties
    Calculations introduce additional event properties
    Grouping by one or more event properties
    Aggregation for each event group over a pre-defined period of time, typically a window
    Multiple event groups monitored by the same query
    Correlate event streams
    Check for absence of activity with a data source
    Enrich events with reference data
    Collection of assets may change over time
    We want to make writing and maintaining those queries easy or even effortless
  • 64. CEP Query Features
    Operators over streams
    Calculations (PROJECT)
    Correlation of streams from different data sources (JOIN)
    Check for absence of activity with a data source (EXISTS)
    Selection of events from streams (FILTER)
    Stream partitioning (GROUP & APPLY)
    Aggregation (SUM, COUNT, …)
    Ranking and heavy hitters (TOP-K)
    Temporal operations: hopping window, sliding window
    Extensibility – to add new domain-specific operators
    Queries are written over specific event types
    They can be evaluated on all data sources with the same event type
    Support for streaming data, reference data (lookup), and historical data (replay)
  • 65. LINQ Query Examples
    wheree1.f2 = “foo”
    select new { e1.f1, e2.f4 };
    groupe3bye3.iinto SubStreams
    select new {
    pl =newMyNewPayload(e4.i, e4.f)};
  • 66. Extensibility
    Built-in operators do not cover all functionality
    Need for domain-specific extensions
    Integrate with functionality available in existing libraries
    Support for extensions in the CEP platform:
    User-defined operators, functions, aggregates
    Code written in .NET, deployed as .NET assembly
    Query operators can refer to functionality of the assembly
    LINQ queries can easily refer to user-defined functionality
    Temporal snap-shot operator framework
    Interface to implement user-defined operators
    Manages operator state and snapshot changes
    Framework does the heavy lifting to deal with intricate temporal behavior such as out-of-order events
  • 67. Developer Experience with the CEP Platform
    Easy to use and highly productive
    Most comprehensive toolset
    Supports various deployment models for CEP applications
    • Embedded applications
    • 68. Server applications
    Compelling story for domain-specific extensions
    CEP development environment
    Builds on top of .NET
    Microsoft Visual Studio as IDE
    Applications written in C#
    Queries written in LINQ
    varqueryFilter =
    wherec.Field1 > 1
  • 69. Managing CEP Data Sources & Queries
    Typical CEP Application
    • Multiple different data sources and adapters
    • 70. Several queries
    Intellectual property is in the queries
    Protect investment in queries over time
    • Re-use existing queries as newer version of an asset, device or data source become available
    • 71. Re-use existing queries for new data source types
    • 72. Seamlessly switch between live data feeds and historical data
  • Query Binding
    Analytics & Queries
    Data Sources
    CEP Engine
    Standing Queries
    Input Adapters
    Output Adapters
    Binding a query with data sources yields a query instance
    Configure multiple query instances for a query
    Similar to data independence: the same query can be bound to different data sources as long as they deliver the required event type
  • 73. CEP Query Configuration
    Query binding:
    Coupling adapters with a query
    Event types required by the query need to be matched by types delivered by adapters
    Query can be re-used for all data sources of the same type
    No changes to query necessary for re-use
    Output Adapter
  • 74. Recap: CEP Platform from Microsoft
    Development experience with .NET, C#, LINQ and Visual Studio 2008
    CEP Application Development
    CEP platform from Microsoft to build event-driven applications
    Event targets
    Event sources
    Event-driven applications are fundamentally different from traditional database applications: queries are continuous, consume and produce streams, and compute results incrementally
    CEP Engine
    Standing Queries
    Output Adapters
    Input Adapters
    Flexible adapter SDK with high performance to connect to different event sources and sinks
    The CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data
    Static reference data
  • 75. CEP Deployment Scenarios
    Custom CEP Application
    Scenario 1: Custom CEP Application Dev
    Scenario 2: Embed CEP in Application
    ISV Application with CEP Engine
    .NET, C#, LINQ
    CEP Engine
    CEP Engine
    KPI mining
    ETL Pipeline with CEP engine
    CEP Engine
    Scenario 4: Operational Intelligence w/ CEP
    Scenario 3: CEP Enabled Device
    Reference data
    Device with Embedded CEP Engine
    Reference data
    CEP Engine
  • 76. CEP Platform Roadmap
    Focus on
    Custom development platform for CEP applications
    CEP platform for Microsoft partners
    TAP (Technology Adopter Program) will launch in July 2009
    Product availability targeted for 2010
    TAP program
    TAP continues
    General Availability
    Late 2009
    July 2009
  • 77. Anot-for-profit, user-run association to enable the SQL Server community to connect, share, and learn
    Get ongoing SQL Server learning from MVPs, top experts, and real-world users:
    • Training webcasts, technical education, worldwide event info, and more
    Sign up for your FREE PASS membership today:
    • Go to local PASS Chapter meetings
    Visit the PASS Chapter booth to learn about upcoming local events, or visit
    • No local Chapter? Start a Chapter with support and resources from PASS
    • Attend PASS Summit 2010
    World's largest and best conference for SQL Server and BI technical education
    Follow / Join us online: Twitter @sqlpass #sqlpass, LinkedIn group, Facebook group
  • 78. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
    The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.