What's New in Toolkits for IBM Streams V4.1

© 2015 IBM Corporation
What’s new in Toolkits
IBM Streams 4.1
Ankit Pasricha
Toolkits Team Lead
ankitp@ca.ibm.com

2 © 2015 IBM Corporation
Important Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL
PURPOSES ONLY.
WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE
INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY
OF ANY KIND, EXPRESS OR IMPLIED.
IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY,
WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.
IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR
OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR
THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT
GOVERNING THE USE OF IBM SOFTWARE.
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion. Information regarding potential
future products is intended to outline our general product direction and it should not
be relied on in making a purchasing decision. The information mentioned regarding
potential future products is not a commitment, promise, or legal obligation to deliver
any material, code or functionality. Information about potential future products may
not be incorporated into any contract. The development, release, and timing of any
future features or functionality described for our products remains at our sole
discretion.
THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.
IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

Agenda
 (New) Spark MLLib Toolkit
 (New) Cybersecurity Toolkit
 (New) Distributed Process Store (DPS) Toolkit
 Messaging Toolkit
 Geospatial Toolkit
 Text Toolkit
 Other updates

 Combines the power of Spark MLLib and real-time streaming capabilities of
Streams
 Allows scoring of real-time streaming data using Spark models
 Github project
 http://ibmstreams.github.io/streamsx.sparkMLLib/
 Support for a number of MLLib models
• Classification
• Linear SVM
• Naive Bayes
• Clustering
• KMeans
• Collaborative Filtering
• Regression
• Isotonic
• Linear
• Logistic
• Tree
• Decision Tree
• Gradient Boosted Trees
• Random Forest
Spark MLLib Toolkit

Streams + Spark Demo
Incidents
Calls for
Service
(911, etc)
311
Code
Violations
Permits
Buildings Apache Spark
MLlib
hdfs
Historical City Data Sets
Model :
Is this call for
service a
false alarm?
Real-time
Calls for Service
Real-time
Predictions &
Relevant Context
IBM
Streams
Real-time
Dashboard

Resources
 Getting Started Guide:
https://developer.ibm.com/streamsdev/docs/getting-started-with-the-
spark-mllib-toolkit/
 Documentation:
http://ibmstreams.github.io/streamsx.sparkMLLib/com.ibm.streamsx.
sparkmllib/doc/spldoc/html/
 MLLib Guide: https://spark.apache.org/docs/latest/mllib-guide.html
 Samples:
https://github.com/IBMStreams/streamsx.sparkMLLib/tree/master/sa
mples

Cybersecurity Toolkit
 The toolkit can detect active threats occurring within a network in
real-time.
 Contains 3 machine-learning cybersecurity models:
 DomainProfiling: Capable of analyzing DNS response records and reporting
on whether any domains are behaving suspiciously
 HostProfiling: Capable of analyzing DNS response records and reporting if
individual hosts are behaving suspiciously
 PredictiveBlacklisting: Capable of analyzing DNS response records and
predicting if a domain should be added to an internal blacklist

Resources
 Introduction to Cybersecurity toolkit:
https://developer.ibm.com/streamsdev/docs/detect-active-threats-in-
real-time-streams-cybersecurity-toolkit/
 Getting Started Guide:
http://ibmstreams.github.io/streamsx.documentation/docs/4.1/cybers
ecurity/cybersecurity-getting-started/
 Documentation: http://www-
01.ibm.com/support/knowledgecenter/SSCRJU_4.1.0/com.ibm.strea
ms.toolkits.doc/spldoc/dita/tk$com.ibm.streams.cybersecurity/tk$co
m.ibm.streams.cybersecurity.html?lang=en
 Starter Apps:
https://github.com/IBMStreams/streamsx.cybersecurity.starterApps

Distributed Process Store (DPS) Toolkit
 Allows sharing of data across operators, Streams applications and
Streams and other applications.
– Provides a collection of APIs in Java, C++ and SPL to read/write from redis
– Support for Redis 2.8.x and 3.0
 Java Example: Creating a distributed store

Distributed Process Store (DPS) Toolkit
 Java Example: Acquiring a distributed lock
 Java Example: Writing data

Resources
 Documentation: https://www-
01.ibm.com/support/knowledgecenter/SSCRJU_4.1.0/com.ibm.strea
ms.toolkits.doc/spldoc/dita/tk$com.ibm.streamsx.dps/tk$com.ibm.st
reamsx.dps.html?lang=en
 Samples:
https://github.com/IBMStreams/streamsx.dps/tree/master/com.ibm.st
reamsx.dps/samples

Messaging Toolkit updates
 Guaranteed Processing Support
 KafkaConsumer
 On checkpoint: save the offset within the message log
 On reset: Replay messages from offset
 JMSSource
 Runs in a transacted session with MQ running in persistent mode
 On checkpoint: Acknowledge read messages so they are removed from the queue
 On reset: Start replaying any unacknowledged messages
 Performance improvements in Kafka operators (Pre-release)
 Using new KafkaProducer API
 Developed in github: https://github.com/IBMStreams/streamsx.messaging
 RabbitMQ support (Pre-release)
 Kafka 0.9 and Message Hub support on Bluemix

Geospatial Toolkit Update
 New PointMapMatcher operator
 We have a map, and a set of imprecise points
coming from a GPS or some other source in real
time
 The data may only have a certain inherent
precision
 There may be errors due signal noise
 The map itself may be imprecise or
incorrect
 We want to clean and smooth this data one
point at a time to lock the incoming points to the
road network.
 “Where is this entity right now?”

Operator Details
14
PointMapMatcher
Entity Locations
Map Geometry Updates
Matches
Errors

15 © 2015 IBM Corporation15
Some use cases:
• Routing
• Traffic reports
• Transit scheduling
• Taxi/emergency dispatching
• Streams Dev article:
https://developer.ibm.com/streamsdev/do
cs/realtime-map-matching-in-streams-v4-
0-1/

Text Toolkit update
 Added support for AQLs generated from BI 4.0+ web tooling
 2 Step process
– Step 1: Create an extractor in BI web tool

Text Toolkit update
 Step 2: Load the extractor in the TextExtract operator for execution
stream<DataToAnalyze, ReferencesFound> TextExtractOutput =
TextExtract(InputFromSocialMedia)
{
param
moduleSearchPath : "etc/extractor" ;
inputDoc : "text" ;
outputViews : "ProductSearch" ;
outputMode : "multiPort" ;
}
 For more information: http://www-
01.ibm.com/support/knowledgecenter/SSCRJU_4.1.0/com.ibm.stream
s.toolkits.doc/spldoc/dita/tk$com.ibm.streams.text/tk$com.ibm.strea
ms.text.html?lang=en

Other updates
 Bluemix support
 HDFS Toolkit for Bluemix
 Hbase Toolkit for Bluemix
 More information: https://developer.ibm.com/streamsdev/docs/integrating-
streams-biginsights-hbase-service-bluemix/
 Data Governance support
 HDFS Toolkit
 DB Toolkit
 Inet Toolkit
 Messaging Toolkit
 Webcast Replay: https://developer.ibm.com/streamsdev/docs/streams-v-4-1-0-
developer-conference-replay/
 Support for BI 4.1
 HDFS Toolkit
 Hbase Toolkit

Questions?

What's New in Toolkits for IBM Streams V4.1

More Related Content

What's hot

Viewers also liked

Similar to What's New in Toolkits for IBM Streams V4.1

More from lisanl

Recently uploaded

What's New in Toolkits for IBM Streams V4.1