Webinar: Future-Proof Your Streaming Analytics Architecture
Mike Gualtieri, Principal Analyst
July 23, 2015
Twitter: @mgualtieri
Anand Venugopal, Product Head - StreamAnalytix
Twitter: @streamanalytix
© 2015 Impetus Technologies
Impetus Introduction
› Mission critical technology solutions since 1996
› Global Leaders are our Big Data clients
› 1600 people – US, India, Global reach
› Unique mix of Big Data products and Services
© 2015 Forrester Research, Inc. Reproduction Prohibited 3
Agenda
› Business need for streaming analytics
› Technology overview and use cases
› Architecture blueprint
› Streaming platforms comparison
› Optimal architecture
› StreamAnaytix approach and benefits
Future-Proof Your Streaming
Analytics Architecture
Mike Gualtieri, Principal Analyst
Twitter: @mgualtieri
#Priority
© 2015 Forrester Research, Inc. Reproduction Prohibited 6
52%
53%
53%
54%
58%
64%
64%
65%
66%
73%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Better leverage big data and analytics in business decision-making
Create a comprehensive strategy for addressing digital technologies like mobile, social
& smart products
Create a comprehensive digital marketing strategy
Better comply with regulations and requirements
Improve differentiation in the market
Increase influence and brand reach in the market
Address rising customer expectations
Improve our ability to innovate
Reduce costs
Improve our products /services
Improve the experience of our customers
Customer experience is a top business priority over the
next 12 months
› Base: 3,005 global data and analytics decision-makers
› Source: Global Business Technographics Data And Analytics Online Survey, 2015
For you For all For segments For you
CRM
Hyper-Personal, Real-Time
Digital Experiences
Personal
Relationships
Mass Production
CustomerExperience
1800 1900 1950 2000 2015
#Celebrity
Customers want and increasingly expect to be
treated like celebrities.
• Use analytics to learn customer
characteristics and behavior
• Detect real-time context
• Adapt applications to serve an
individual customer
Celebrity experiences must:
Be Blazing Fast
#BigData
© 2015 Forrester Research, Inc. Reproduction Prohibited 12
All data starts out fast, but is often only used after it
becomes big data at-rest…
› Real-time transactional data from portfolio of dozens or hundreds of
business applications
› Real-time usage and behavior data from web and mobile apps
› Real-time social media data
› Real-time IoT data from sensors and devices
› Real-time data services that sell data
…but, that’s not good enough to create modern apps.
#Analytics
© 2015 Forrester Research, Inc. Reproduction Prohibited 14
Three kinds of analytics are essential to create
applications that deliver celebrity experiences.
Past Present Future
Learn Contextualize Predict
Predictive
Analytics
Streaming
Analytics
Historical Analytics
(Traditional Analytics)
(Advanced Analytics)
15© 2015 Forrester Research, Inc. Reproduction Prohibited
Source: Forrester Research
Streaming analytics is among the hottest of advanced
analytics technologies
39%
42%
42%
42%
42%
43%
43%
46%
48%
52%
54%
55%
56%
57%
69%
Non Modeled Data Exploration And Discovery
Search/Interactive Discovery
Streaming Analytics
Metadata Generated Analytics
OLAP
Advanced Visualization
Text Analytics
Location Analytics
Predictive Analytics
Process Analytics
Embedded Analytics
Web Analytics
Dashboards
Performance analytics
Reporting
2015
2014
“What is your firm's/business unit's current use of the following technologies?”
Source: Forrester's Global Business Technographics Data And Analytics Survey, 2015 and 2014
Base: 1805 (2015), 1063 (2014)
© 2015 Forrester Research, Inc. Reproduction Prohibited 16
Real-time means business time
› A customer walks into a shopping mall
› A shopper clicks on an online add
› A temperature sensor spikes
› A stock price rises
› A customer uses a credit card
› A customer wakes up
Streaming data is flowing by, and opportunity
is slipping away.
#Streaming
Blazing fast ingestion, analysis, and actions on
multiple sources of fast data.
DEFINITION
FORRESTERStreaming analytics platforms can filter,
aggregate, enrich, and analyze a high
throughput of data from disparate live data
sources to identify patterns, detect urgent
situations, and automate immediate actions.
© 2015 Forrester Research, Inc. Reproduction Prohibited 21
Thinking in streams is very different from traditional
historical analytics
› Filtering
› Aggregation/correlation
› Enrichment
› Time windows
› Temporal patterns
› Rules
› Scoring
› Computation
› Location/motion
› Query and action interfaces
How can huge volume of
telematics data from 250+
onboard sensors be used to
improve safety?
Capture and analyze all
data to predict part
failures.
Fortune 10 technology company.
How can a mobile travel
app predict the users next
desire.
Use real-time location
analytics.
How can a farm equipment
company predict failures to
reduce maintenance cost
and increase uptime.
Capture and analyze IoT
data in real-time.
How can a flash-sale online
retailer predict what
customers will buy and how
much they will pay?
Continuously monitor
customer behavior to adapt
prices and catalog in real-
time.
How can a media company
show the most relevant ad.
Detect who is watching
the TV in the household
in real-time.
#Architecture
Streaming
Real-Time
Data
Sources
Scale should not limit design decisions.
Fault-tolerance is non-negotiable.
Streaming must fit and work seamlessly with
existing architectures.
Accommodate analytical and transactional
streams of data.
Have the horsepower to perform real-time analytics.
Streaming technology speeds application
development by reducing architectural concerns.
#Requirements
© 2015 Forrester Research, Inc. Reproduction Prohibited 36
Streaming analytics platforms evaluation criteria (1 of 3)
› Architecture
• Runtime deployment options
• Performance and scalability
• High-availability and disaster recovery
• Setup, management, and monitoring tools
• Security
› Data Sources
• Input/ Output
© 2015 Forrester Research, Inc. Reproduction Prohibited 37
Streaming analytics platforms evaluation criteria (2 of 3)
› Development tools
• Professional development
• Business development
• Application dev lifecycle tools
• Debugging
• Testing
• Simulation
© 2015 Forrester Research, Inc. Reproduction Prohibited 38
Streaming analytics platforms evaluation criteria (3 of 3)
› Stream processing operators (built-in)
• Filtering
• Aggregation
• Location-based
• Time windows
• Temporal patterns
• Continuous query
• Enrichment
• Action interfaces
• Dynamic operators
• Built-in libraries
• Third-party libraries
• Custom libraries
#Solutions
© 2015 Forrester Research, Inc. Reproduction Prohibited 40
Be careful not to confuse streaming analytics with
these related/complimentary technologies
› Ingestion technologies
• Connections and routing or data
› ETL technologies
• Data transformation targeting at-rest
analytics platforms such as data
warehouse and Hadoop
› Complex event processing (CEP)
• A feature of some streaming analytics
platforms for very low latency pattern
detection
› Real-time dashboards
• View the results of streaming analytics
› Micro services
• Simple stateless event driven processing
› Domain-specific solutions
• Packaged software solutions such as
manufacturing control systems
© 2015 Forrester Research, Inc. Reproduction Prohibited 41
› Big enterprise software vendors
• Pro: Proven, industrial-strength solutions
that include enterprise integration and
tooling
• Con: Cost and complexity of
implementation
› Open source streaming projects
• Pro: Free to use and increasing number of
projects to choose from
• Con: Lacks enterprise dev/management
tools and community consensus
The market streaming analytics platforms is dynamic.
› Streaming analytics startups
• Pro: Purpose-built for today’s diverse
streaming analytics apps and often leverages
open source with some enterprise tooling
• Con: Still early in the market cycle to predict
adoption of solution
› Cloud-exclusive streaming
analytics services
• Pro: Pay-per-use model and integration with
other cloud services
• Con: Provides only limited analytical
operators and lacks on-premise solution
© 2015 Forrester Research, Inc. Reproduction Prohibited 42
Abstract your streaming analytics capability to future-
proof your solution
Ingest Prepare Analyze Decide Act
Streaming Data
Sensors
Social
Machine Data
Location
Transactions
Logs
Transform
Filter
Correlate
Aggregate
Enrich
Classification
Patterns
Anomalies
Scoring
Events
Computation
Rules
Logic
Policies
Notify
Publish
Execute
Visualize
© 2015 Impetus Technologies
The 3rd Approach: Best of Both Worlds
StreamAnalytix mitigates the
disadvantages of the
"default"
approaches and offers the
benefits
of both worlds to enterprises
for streaming analytics.
Abstraction of functional components like Ingest, CEP, Analytics, Visualization
© 2015 Impetus Technologies
STORM SPARK OTHERS
NOW
Time
Abstraction of Technologies
StreamAnalytix gives you a future proof option
© 2015 Impetus Technologies
Real-time streaming analytics platform
› Why ?
• Customers, Operations
• Build Vs. Buy
› What to buy ?
• Architecture requirements, Abstraction
• Integrated architecture
› From whom to buy ? and…What to watch out for ?
• Time to market and long term value
© 2015 Impetus Technologies
Context aware  positive customer experience
Multi-channel
engagement in real-
time
Context
Sensitive service
Happy customers,
Loyalty, Revenue,
Profits, Growth
© 2015 Impetus Technologies
Batch vs. Real-time business process
SENSE Days ANALYSE Weeks ACT
SENSE ANALYSE ACT
Sec/ ms
Batch
Real time
Sec/ ms
© 2015 Impetus Technologies
t
now
Hadoop works great back here RT-Ax works
here
Blended view – historical and now
Blended viewBlended viewBlended View
© 2015 Impetus Technologies
Lambda architecture : big and fast data combined
Batch Layer
All data
Pre-computed
information
Batch re-compute
Speed Layer
All data
Pre-computed
information
Real time increment
Batch view
Serving Layer
Batch view
Merge
Real time view
Real time view
All
Incoming
Data Query
© 2015 Impetus Technologies
An integrated approach blending current and next generation tech
Landing and
ingestion
Structured
Unstructured
External
Social
Machine
Geospatial
Time Series
Streaming
Provisioning, Workflow, Monitoring and Security
Enterprise
Data Lake
Predictive
applications
Exploration & discovery
Enterprise
applications
Real-time applications
Traditional
data repositories
RDBMS MPP
Compliance, Governance, Information Lifecycle, Data Lineage, Enterprise Meta
Data Management
© 2015 Impetus Technologies
Future proof – Enterprise Grade – Open source based – Streaming Analytics platform
NEXT
Unified Business Interfaces Common Utilities Smart Workflows
© 2015 Impetus Technologies
Poll:
› YES
› NOT PARTICULARLY
Is an architecture that offers functional and technology abstraction for
Streaming Analytics with the required scale and performance attractive
to you from an evaluation perspective ?
© 2015 Impetus Technologies
From whom to buy ? IMPETUS
› Right size
› Independent
› Services
› Track record of Long term partnerships and value
› Recent success stories
?
Q&A
(Use the chat/Q&A panel)
inquiry@streamanalytix.com
www.StreamAnalytix.com
?
Contact us for an On-premise OR Cloud based trial and/or Proof of concept
Meet us at Strata Hadoop World in New York in September

Future-Proof Your Streaming Analytics Architecture- StreamAnalytix Webinar

  • 1.
    Webinar: Future-Proof YourStreaming Analytics Architecture Mike Gualtieri, Principal Analyst July 23, 2015 Twitter: @mgualtieri Anand Venugopal, Product Head - StreamAnalytix Twitter: @streamanalytix
  • 2.
    © 2015 ImpetusTechnologies Impetus Introduction › Mission critical technology solutions since 1996 › Global Leaders are our Big Data clients › 1600 people – US, India, Global reach › Unique mix of Big Data products and Services
  • 3.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 3 Agenda › Business need for streaming analytics › Technology overview and use cases › Architecture blueprint › Streaming platforms comparison › Optimal architecture › StreamAnaytix approach and benefits
  • 4.
    Future-Proof Your Streaming AnalyticsArchitecture Mike Gualtieri, Principal Analyst Twitter: @mgualtieri
  • 5.
  • 6.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 6 52% 53% 53% 54% 58% 64% 64% 65% 66% 73% 75% 0% 10% 20% 30% 40% 50% 60% 70% 80% Better leverage big data and analytics in business decision-making Create a comprehensive strategy for addressing digital technologies like mobile, social & smart products Create a comprehensive digital marketing strategy Better comply with regulations and requirements Improve differentiation in the market Increase influence and brand reach in the market Address rising customer expectations Improve our ability to innovate Reduce costs Improve our products /services Improve the experience of our customers Customer experience is a top business priority over the next 12 months › Base: 3,005 global data and analytics decision-makers › Source: Global Business Technographics Data And Analytics Online Survey, 2015
  • 7.
    For you Forall For segments For you CRM Hyper-Personal, Real-Time Digital Experiences Personal Relationships Mass Production CustomerExperience 1800 1900 1950 2000 2015
  • 8.
  • 9.
    Customers want andincreasingly expect to be treated like celebrities.
  • 10.
    • Use analyticsto learn customer characteristics and behavior • Detect real-time context • Adapt applications to serve an individual customer Celebrity experiences must: Be Blazing Fast
  • 11.
  • 12.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 12 All data starts out fast, but is often only used after it becomes big data at-rest… › Real-time transactional data from portfolio of dozens or hundreds of business applications › Real-time usage and behavior data from web and mobile apps › Real-time social media data › Real-time IoT data from sensors and devices › Real-time data services that sell data …but, that’s not good enough to create modern apps.
  • 13.
  • 14.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 14 Three kinds of analytics are essential to create applications that deliver celebrity experiences. Past Present Future Learn Contextualize Predict Predictive Analytics Streaming Analytics Historical Analytics (Traditional Analytics) (Advanced Analytics)
  • 15.
    15© 2015 ForresterResearch, Inc. Reproduction Prohibited Source: Forrester Research Streaming analytics is among the hottest of advanced analytics technologies 39% 42% 42% 42% 42% 43% 43% 46% 48% 52% 54% 55% 56% 57% 69% Non Modeled Data Exploration And Discovery Search/Interactive Discovery Streaming Analytics Metadata Generated Analytics OLAP Advanced Visualization Text Analytics Location Analytics Predictive Analytics Process Analytics Embedded Analytics Web Analytics Dashboards Performance analytics Reporting 2015 2014 “What is your firm's/business unit's current use of the following technologies?” Source: Forrester's Global Business Technographics Data And Analytics Survey, 2015 and 2014 Base: 1805 (2015), 1063 (2014)
  • 16.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 16 Real-time means business time › A customer walks into a shopping mall › A shopper clicks on an online add › A temperature sensor spikes › A stock price rises › A customer uses a credit card › A customer wakes up
  • 17.
    Streaming data isflowing by, and opportunity is slipping away.
  • 18.
  • 19.
    Blazing fast ingestion,analysis, and actions on multiple sources of fast data.
  • 20.
    DEFINITION FORRESTERStreaming analytics platformscan filter, aggregate, enrich, and analyze a high throughput of data from disparate live data sources to identify patterns, detect urgent situations, and automate immediate actions.
  • 21.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 21 Thinking in streams is very different from traditional historical analytics › Filtering › Aggregation/correlation › Enrichment › Time windows › Temporal patterns › Rules › Scoring › Computation › Location/motion › Query and action interfaces
  • 22.
    How can hugevolume of telematics data from 250+ onboard sensors be used to improve safety? Capture and analyze all data to predict part failures.
  • 23.
    Fortune 10 technologycompany. How can a mobile travel app predict the users next desire. Use real-time location analytics.
  • 24.
    How can afarm equipment company predict failures to reduce maintenance cost and increase uptime. Capture and analyze IoT data in real-time.
  • 25.
    How can aflash-sale online retailer predict what customers will buy and how much they will pay? Continuously monitor customer behavior to adapt prices and catalog in real- time.
  • 26.
    How can amedia company show the most relevant ad. Detect who is watching the TV in the household in real-time.
  • 27.
  • 28.
  • 29.
    Scale should notlimit design decisions.
  • 30.
  • 31.
    Streaming must fitand work seamlessly with existing architectures.
  • 32.
    Accommodate analytical andtransactional streams of data.
  • 33.
    Have the horsepowerto perform real-time analytics.
  • 34.
    Streaming technology speedsapplication development by reducing architectural concerns.
  • 35.
  • 36.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 36 Streaming analytics platforms evaluation criteria (1 of 3) › Architecture • Runtime deployment options • Performance and scalability • High-availability and disaster recovery • Setup, management, and monitoring tools • Security › Data Sources • Input/ Output
  • 37.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 37 Streaming analytics platforms evaluation criteria (2 of 3) › Development tools • Professional development • Business development • Application dev lifecycle tools • Debugging • Testing • Simulation
  • 38.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 38 Streaming analytics platforms evaluation criteria (3 of 3) › Stream processing operators (built-in) • Filtering • Aggregation • Location-based • Time windows • Temporal patterns • Continuous query • Enrichment • Action interfaces • Dynamic operators • Built-in libraries • Third-party libraries • Custom libraries
  • 39.
  • 40.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 40 Be careful not to confuse streaming analytics with these related/complimentary technologies › Ingestion technologies • Connections and routing or data › ETL technologies • Data transformation targeting at-rest analytics platforms such as data warehouse and Hadoop › Complex event processing (CEP) • A feature of some streaming analytics platforms for very low latency pattern detection › Real-time dashboards • View the results of streaming analytics › Micro services • Simple stateless event driven processing › Domain-specific solutions • Packaged software solutions such as manufacturing control systems
  • 41.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 41 › Big enterprise software vendors • Pro: Proven, industrial-strength solutions that include enterprise integration and tooling • Con: Cost and complexity of implementation › Open source streaming projects • Pro: Free to use and increasing number of projects to choose from • Con: Lacks enterprise dev/management tools and community consensus The market streaming analytics platforms is dynamic. › Streaming analytics startups • Pro: Purpose-built for today’s diverse streaming analytics apps and often leverages open source with some enterprise tooling • Con: Still early in the market cycle to predict adoption of solution › Cloud-exclusive streaming analytics services • Pro: Pay-per-use model and integration with other cloud services • Con: Provides only limited analytical operators and lacks on-premise solution
  • 42.
    © 2015 ForresterResearch, Inc. Reproduction Prohibited 42 Abstract your streaming analytics capability to future- proof your solution Ingest Prepare Analyze Decide Act Streaming Data Sensors Social Machine Data Location Transactions Logs Transform Filter Correlate Aggregate Enrich Classification Patterns Anomalies Scoring Events Computation Rules Logic Policies Notify Publish Execute Visualize
  • 43.
    © 2015 ImpetusTechnologies The 3rd Approach: Best of Both Worlds StreamAnalytix mitigates the disadvantages of the "default" approaches and offers the benefits of both worlds to enterprises for streaming analytics. Abstraction of functional components like Ingest, CEP, Analytics, Visualization
  • 44.
    © 2015 ImpetusTechnologies STORM SPARK OTHERS NOW Time Abstraction of Technologies StreamAnalytix gives you a future proof option
  • 45.
    © 2015 ImpetusTechnologies Real-time streaming analytics platform › Why ? • Customers, Operations • Build Vs. Buy › What to buy ? • Architecture requirements, Abstraction • Integrated architecture › From whom to buy ? and…What to watch out for ? • Time to market and long term value
  • 46.
    © 2015 ImpetusTechnologies Context aware  positive customer experience Multi-channel engagement in real- time Context Sensitive service Happy customers, Loyalty, Revenue, Profits, Growth
  • 47.
    © 2015 ImpetusTechnologies Batch vs. Real-time business process SENSE Days ANALYSE Weeks ACT SENSE ANALYSE ACT Sec/ ms Batch Real time Sec/ ms
  • 48.
    © 2015 ImpetusTechnologies t now Hadoop works great back here RT-Ax works here Blended view – historical and now Blended viewBlended viewBlended View
  • 49.
    © 2015 ImpetusTechnologies Lambda architecture : big and fast data combined Batch Layer All data Pre-computed information Batch re-compute Speed Layer All data Pre-computed information Real time increment Batch view Serving Layer Batch view Merge Real time view Real time view All Incoming Data Query
  • 50.
    © 2015 ImpetusTechnologies An integrated approach blending current and next generation tech Landing and ingestion Structured Unstructured External Social Machine Geospatial Time Series Streaming Provisioning, Workflow, Monitoring and Security Enterprise Data Lake Predictive applications Exploration & discovery Enterprise applications Real-time applications Traditional data repositories RDBMS MPP Compliance, Governance, Information Lifecycle, Data Lineage, Enterprise Meta Data Management
  • 51.
    © 2015 ImpetusTechnologies Future proof – Enterprise Grade – Open source based – Streaming Analytics platform NEXT Unified Business Interfaces Common Utilities Smart Workflows
  • 52.
    © 2015 ImpetusTechnologies Poll: › YES › NOT PARTICULARLY Is an architecture that offers functional and technology abstraction for Streaming Analytics with the required scale and performance attractive to you from an evaluation perspective ?
  • 53.
    © 2015 ImpetusTechnologies From whom to buy ? IMPETUS › Right size › Independent › Services › Track record of Long term partnerships and value › Recent success stories ?
  • 54.
    Q&A (Use the chat/Q&Apanel) inquiry@streamanalytix.com www.StreamAnalytix.com ? Contact us for an On-premise OR Cloud based trial and/or Proof of concept Meet us at Strata Hadoop World in New York in September