Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
IOT, STREAMING ANALYTICS AND MACHINE LEARNING
Delivering Real-Time Intelligence With Apache NiFi
Paul Kent, VP of Big Data...
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
• Drop-and-Drag Interface
• Secure/Encrypted
• Bi-Directional ...
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
SAS ESP + HDF WHY IS THIS IMPORTANT?
RAPID PROTOTYPING OF MACH...
Events
(Data In)
Events
(Data Out)
Filtering
Aggregation
Pattern Detection
Computation
Merging / Joins
Functions
Retention...
KEY CONCEPTSESP MODEL - PROCESS FLOW
SAS EVENT STREAM PROCESSING ENGINE
DATA IN
(Events)
DATA OUT
(Events)
Design of the r...
Copyr ight © 2012, SAS Institute Inc. All rights reser ved.
EVENT STREAM
PROCESSING
ESP STUDIO INTERFACE
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
DESIGNING ESP
MODELS
DESIGN COMPONENTS
DS2 PROCEDURAL WINDOW
I...
Copyr ight © 2012, SAS Institute Inc. All rights reser ved.
DESIGNING ESP
MODELS
DESIGN COMPONENTS
PATTERN WINDOW overview...
Copyr ight © 2012, SAS Institute Inc. All rights reser ved.
DESIGNING ESP
MODELS
DESIGN COMPONENTS
TEXT ANALYTICS WINDOWS
...
SAS EVENT STREAM
PROCESSING
HTML5 STREAMVIEWER
• HTML5 interface
• Uses HTTP (RESTful) XML server
• 2 Modes:
• Streaming m...
INTEGRATION
SAS EVENT STREAM PROCESSING &
HORTONWORKS DATA FLOW (NIFI)
&
SAS Event Stream Processing Hortonworks Data Flow...
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
SAS®
EVENT STREAM
PROCESSING CONCEPTUAL OVERVIEW
SAS-generated...
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
SAS EVENT STREAM
PROCESSING
• CONNECTORS & ADAPTERS
PUB/SUB AP...
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
DIRECTION: ADAPTERS & CONNECTORS
•Flume: Integrate ESP with st...
Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
SAS® EVENT STREAM
PROCESSING 4.1
FLEXIBILITY AND INTEGRATION
•...
Company Confidential - For Internal Use Only
Copyright © 2015, SAS Institute Inc. All rights reserved.
Edge
Analytics
In-M...
STREAMING
ANALYTICS
Where are the Opportunities?
• Competitive Pressure (Technology, Sensors, Analytics)
• Risk
• Safety
•...
INTEGRATION
SAS EVENT STREAM PROCESSING &
HORTONWORKS DATA FLOW (NIFI)
&
SAS Event Stream Processing Hortonworks Data Flow...
Copyright © 2012, SAS Institute Inc. All rights reserved.
Paul Kent, VP of Big Data, Platform R&D
Dan Zaratsian, Sr. Solut...
Company Confidential - For Internal Use Only
Copyright © 2015, SAS Institute Inc. All rights reserved.
Company Confidential - For Internal Use Only
Copyright © 2015, SAS Institute Inc. All rights reserved.
Company Confidential - For Internal Use Only
Copyright © 2015, SAS Institute Inc. All rights reserved.
Company Confidential - For Internal Use Only
Copyright © 2015, SAS Institute Inc. All rights reserved.
Upcoming SlideShare
Loading in …5
×

IOT, Streaming Analytics and Machine Learning

2,870 views

Published on

IOT, Streaming Analytics and Machine Learning

Published in: Technology
  • Be the first to comment

IOT, Streaming Analytics and Machine Learning

  1. 1. IOT, STREAMING ANALYTICS AND MACHINE LEARNING Delivering Real-Time Intelligence With Apache NiFi Paul Kent, VP of Big Data, Platform R&D Dan Zaratsian, Sr. Solutions Architect
  2. 2. Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
  3. 3. Copyr ight © 2015, SAS Institute Inc. All rights reser ved. • Drop-and-Drag Interface • Secure/Encrypted • Bi-Directional Communication • Data Provenance
  4. 4. Copyr ight © 2015, SAS Institute Inc. All rights reser ved.
  5. 5. Copyr ight © 2015, SAS Institute Inc. All rights reser ved. SAS ESP + HDF WHY IS THIS IMPORTANT? RAPID PROTOTYPING OF MACHINE LEARNING MODELS ANALYTICS WITHIN AN OPEN FRAMEWORK
  6. 6. Events (Data In) Events (Data Out) Filtering Aggregation Pattern Detection Computation Merging / Joins Functions Retention Window Text Analytics Unsupervised Learning Predictive Modeling …more… Detect Events & Patterns of Interest
  7. 7. KEY CONCEPTSESP MODEL - PROCESS FLOW SAS EVENT STREAM PROCESSING ENGINE DATA IN (Events) DATA OUT (Events) Design of the rule model (called “Continuous Query”) using components (called “Windows”) DATA IN (Events) DATA IN (Events) DATA OUT (Events) SOURCE 1 WINDOW SOURCE 2 WINDOW SOURCE 3 WINDOW FILTER WINDOW CALCULATIONS WINDOW JOIN WINDOW JOIN WINDOW NOTIFICATION WINDOW PREDICTIVE MODEL (SCORING) WINDOW
  8. 8. Copyr ight © 2012, SAS Institute Inc. All rights reser ved. EVENT STREAM PROCESSING ESP STUDIO INTERFACE
  9. 9. Copyr ight © 2015, SAS Institute Inc. All rights reser ved. DESIGNING ESP MODELS DESIGN COMPONENTS DS2 PROCEDURAL WINDOW In-Stream Analytics: 1. Build analytical model using EM,VA, etc. • Decision Tree • Neural Network • Regression • Rule Induction • And more 2. Use PROC DSTRANS to convert code to DS2 3. Deploy model to procedural window Only when the existing model is additive in nature and can process one event at a time.
  10. 10. Copyr ight © 2012, SAS Institute Inc. All rights reser ved. DESIGNING ESP MODELS DESIGN COMPONENTS PATTERN WINDOW overview Build complex network of events using temporal conditions Multiple events in can produce one event out E1 E2 And Followed By E4 E5 And Not E6 E3Or 5 min 1 hour Followed By “Detect when event A is followed by event B and not Event C in a 3min time frame”
  11. 11. Copyr ight © 2012, SAS Institute Inc. All rights reser ved. DESIGNING ESP MODELS DESIGN COMPONENTS TEXT ANALYTICS WINDOWS • Process unstructured text fields • 3 dedicated Text Analytics windows • Text Context (.liti files) • Text Category (.mco files) • Text Sentiment (.sam files) • An appropriate Text Analytics license is required.
  12. 12. SAS EVENT STREAM PROCESSING HTML5 STREAMVIEWER • HTML5 interface • Uses HTTP (RESTful) XML server • 2 Modes: • Streaming mode: display all events • Update : events processed with opcode • Google charts • Subscribe & Publish
  13. 13. INTEGRATION SAS EVENT STREAM PROCESSING & HORTONWORKS DATA FLOW (NIFI) & SAS Event Stream Processing Hortonworks Data Flow (Nifi)
  14. 14. Copyr ight © 2015, SAS Institute Inc. All rights reser ved. SAS® EVENT STREAM PROCESSING CONCEPTUAL OVERVIEW SAS-generated Insights Enrichment Data Event Actions SAS In-Memory SAS ® Event Stream Processing Model Continuous Query Publish Subscribe Streaming Events Analytic Models Business Rules Nifi
  15. 15. Copyr ight © 2015, SAS Institute Inc. All rights reser ved. SAS EVENT STREAM PROCESSING • CONNECTORS & ADAPTERS PUB/SUB API Connect to any system with Java or C Public, documented and easy to use Adapters are standalone processes and can be networked Publish to ESP Source windows – Subscribe to any ESP window All Connectors & Adapters are built using the Pub/Sub API •File/Socket •XML / JSON •Database (odbc) •SAS® LASR™ •Hadoop •SAS® Dataset OUT OF THE BOX *Publish only **Subscribe only •ESP Project •RabbitMQ •Solace •Tervela •Google Protobuff •Twitter* •SAS® HDAT •JMS •IBM WebSphere MQ •Tibco RendezVous •Syslog * •Network Sniffer* •HTTP RESTful •OSIsoft PI •Axeda •Teradata •SMTP ** •ESP to ESP
  16. 16. Copyr ight © 2015, SAS Institute Inc. All rights reser ved. DIRECTION: ADAPTERS & CONNECTORS •Flume: Integrate ESP with streaming log data •Kafka: Integration with large scale message processing •MQTT: Support within IoT and Connected things •Cassandra (adaptor only): integration with large-scale, distributed data source •HortonWorks Data Flow (NiFi) Processor: support NiFI streams •MapR: MapR Streams support •Boardreader: Blogs, News, Boards, Reviews •Spryware: Market data through direct exchange feeds •IOT Gateways and devices PUB/SUB API Connect to any system with Java or C Documented and easy to use Adapters are standalone processes and can be networked Publish to ESP Source windows – Subscribe to any ESP window All Connectors & Adapters are built using the Pub/Sub API SAS® EVENT STREAM PROCESSING 4.1
  17. 17. Copyr ight © 2015, SAS Institute Inc. All rights reser ved. SAS® EVENT STREAM PROCESSING 4.1 FLEXIBILITY AND INTEGRATION • Python Pub/sub API: drive ESP using Python • Leverage Analytic Decisions within ESP • Decision/Rules/Analytical Model Integration via SAS Micro Analytic Service • Lightweight, fast service for decision deployment • Leverage Languages in ESP (In-process Event Stream Handlers) • DATAstep (native) • DS2 (current) • Python • Future: R language (post-16w48)
  18. 18. Company Confidential - For Internal Use Only Copyright © 2015, SAS Institute Inc. All rights reserved. Edge Analytics In-Motion Analytics At-Rest Analytics Connected Systems, Devices Monitor equipment on for failures and safety issues, and take action. Identify fraudulent transactions and be alerted in real-time. Intelligently integrate customer information with real-time streaming data Strategic Data IntegrationTransactions, Logs, Clickstreams
  19. 19. STREAMING ANALYTICS Where are the Opportunities? • Competitive Pressure (Technology, Sensors, Analytics) • Risk • Safety • Security • Personalization Extend the existing analytical footprint! Capture value otherwise lost through information lag
  20. 20. INTEGRATION SAS EVENT STREAM PROCESSING & HORTONWORKS DATA FLOW (NIFI) & SAS Event Stream Processing Hortonworks Data Flow (Nifi)
  21. 21. Copyright © 2012, SAS Institute Inc. All rights reserved. Paul Kent, VP of Big Data, Platform R&D Dan Zaratsian, Sr. Solutions Architect Copyright © 2012, SAS Institute Inc. All rights reserved.Copyright © 2012, SAS Institute Inc. All rights reserved. Demo
  22. 22. Company Confidential - For Internal Use Only Copyright © 2015, SAS Institute Inc. All rights reserved.
  23. 23. Company Confidential - For Internal Use Only Copyright © 2015, SAS Institute Inc. All rights reserved.
  24. 24. Company Confidential - For Internal Use Only Copyright © 2015, SAS Institute Inc. All rights reserved.
  25. 25. Company Confidential - For Internal Use Only Copyright © 2015, SAS Institute Inc. All rights reserved.

×