Streams GitHub Products Overview for IBM InfoSphere Streams V4.0

© 2015 IBM Corporation
Github Projects Overview
IBM InfoSphere Streams 4.0
Samantha Chan
Team Lead, Streams Toolkits Team
For questions about this presentation contact: chanskw@ca.ibm.com

2 © 2015 IBM Corporation
Important Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL
PURPOSES ONLY.
WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE
INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY
OF ANY KIND, EXPRESS OR IMPLIED.
IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY,
WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.
IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR
OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR
THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT
GOVERNING THE USE OF IBM SOFTWARE.
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion. Information regarding potential
future products is intended to outline our general product direction and it should not
be relied on in making a purchasing decision. The information mentioned regarding
potential future products is not a commitment, promise, or legal obligation to deliver
any material, code or functionality. Information about potential future products may
not be incorporated into any contract. The development, release, and timing of any
future features or functionality described for our products remains at our sole
discretion.
THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.
IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

Agenda
 IBMStreams Organization
 Github Projects Overview
 Github Releases
 Support
 How to get a toolkit?
 Propose a new feature / Report a bug!
 Propose a new project
 How to participate?

IBMStreams
https://github.com/IBMStreams
 Open-source organization established on Github in March 2014
 Goals:
– Provide a platform and foster a community to extend and share Streams
programming resources (toolkits, samples, performance benchmark,
utilities, etc.)
– Allow us to deliver new toolkit functions in a more open, agile and rapid
manner
– Improve visibility of Streams programming resources and make them more
easily accessible.

IBMStreams
 March 2014
– Started the 3 repositories:
• 3 toolkits from the Streams product
(HDFS, Messaging, Inet)
 April 2015
– 6 Adapters
– 5 Parsers and Formatters
– 5 Processing and Analytics
– 4 Utilities
– 5 Samples and Demo
– 6 Under Construction
– Total: 31 Projects

Github Projects Overview - Adapters
 These toolkits are included as part of Streams product.
 HBase Toolkit (streamsx.hbase)
– The HBase toolkit provides support for interacting with Apache HBase systems from
InfoSphere Systems. We provide support for reading and writing to HBase.
 HDFS Toolkit (streamsx.hdfs)
– The HDFS Toolkit provides operators that can read and write data from Hadoop
Distributed File System (HDFS).
 Messaging Toolkit (streamsx.messaging)
– The Messaging Toolkit provides operators that allow your Streams application to read
and send messages to popular messaging systems, like Kafka, MQTT, Websphere MQ
and Apache MQ.
 Inet Toolkit (streamsx.inet)
– The Inet toolkit provides support for common internet protocols. Supported protocols
include FTP, WebSocket, HTTP.

Github Projects Overview - Adapters
 These toolkits are only hosted on Github
 Multi-Connection TCP Server Toolkit (streamsx.tcp)
– This toolkit contains a TCPServer operator which allows for multi-
connections, and is a multi-threaded source operator. The operator accepts
text or binary data from one or more TCP sockets.
 Mongo DB Toolkit (streamsx.mongoDB)
– This toolkit provides support for insert and query support for MongoDB

Github Projects Overview – Parsers and Formatters
 JSON Toolkit (streamsx.json)
– The JSON toolkit allows you to convert data from JSON to Streams tuples
format, and vice versa.
 Document Toolkit (streamsx.document)
– This toolkit allows extract text and metadata from documents in a binary
formats such as PDF, Word, Office, etc. For this purpose the toolkit implements
a DocumentSource operator. Some of the supported text extractors
are: Apache Tika, PDFBox, TrueZip, JUnrar, Plain Text
 Bytes Toolkit (streamsx.bytes)
– This toolkit is for ease developing and analysis of binary data. It provides
functions to process string data: ASCII to HEX, HEX to BIN, etc. It is also able
to extract raw string from binary data.

Github Projects Overview – Parsers and Formatters
 Thrift Toolkit (streamsx.thrift)
– This toolkit provides Thrift server and client functionality.
 Adaptive Parser (streamsx.adaptiveParser)
– Repository is created for hosting operators for parsing structured text data
easily. The goal of this project is to ease the parsing, tuple structure definition,
and tuple mapping development steps in a Streams application.

Github Projects Overview – Analytics and Processing
 NgramHashing Toolkit (streamsx.ngrams)
– This toolkit is intended to be a supplement to a wide range of algorithms for
calculating n-grams, such as NLP, machine learning, speech recognition,
compression, etc.
 Regex Toolkit (streamsx.regex)
– This toolkit provides support for the RE2 regular expression library.
 Math Toolkit (streamsx.math)
– This repository contains operators and functions for complex mathematics
and statistics.

Github Projects Overview – Analytics and Processing
 Date Time Toolkit (streamsx.datetime)
– This toolkit contains additional operators and functions to process dates and
times in data.
 Transportation Toolkit (streamsx.transportation)
– It is intended to contain adapters to access transit feed as well as generic
transportation based operators and functions. Initial contribution provides
support to access live bus feeds from NextBus.

Github Projects Overview – Utilities
 Process Store Toolkit (streamsx.ps)
– The Process Store toolkit provides a simple way for the SPL and C++
operators that are fused inside a single PE to share application specific state
information. It does this via a collection of APIs that can be called from any
part of the and C++ operator code.
 Distributed Process Store Toolkit (streamsx.dps)
– The Distributed Process Store toolkit provides a simple way for the SPL, C++
and Java operators belonging to a single or multiple applications to share
application state information via an external key-value store. Some of the
supported value stores systems are: Memcached, Redis, Cassandra, IBM
Cloudant, HBase, Mongo, Couchbase and Aerospike.

Github Projects Overview – Utilities
 Plumbing Toolkit (streamsx.plumbing)
– This toolkit contains an operator that allows your application to
dynamically manipulate application tuple flow to achieve best performance.
 Utilities Project (streamsx.utility)
– This repository contains useful utilities for InfoSphere Streams. For example,
the repository currently has an utility that displays CPU utilization for PEs in a
Streams Instance.

Github Projects Overview – Samples and Demos
 Samples
– This project contains a set of useful sample Streams applications. E.g.
consistent region, Geospatial samples, HDFS samples, Timeseries
samples.
 Benchmark (benchmarks)
– This repository contains performance benchmark applications for
Streams. The project contains the two email processing applications, one is
written with InfoSphere Streams, and another one is written with Apache
Storm.
– These two benchmarks were used as part of a detailed performance
report: https://developer.ibm.com/streamsdev/wp-
content/uploads/sites/15/2014/04/Streams-and-Storm-April-2014-Final.pdf
 Accelerator Demo Project (streamsx.demo.accelerator)
– This repository contains a collection of demo streaming applications for
analyzing smart phone accelerometers or gyroscope data.
– https://developer.ibm.com/streamsdev/docs/streaming-realtime-
smartphone-data-infosphere-streams/

Github Projects Overview – Samples and Demos
 Resource Manager Project (resourceManagers)
– This repository contains projects on getting Streams to work with other
resource managers, like Yarn.
 Log Watch Demo (streamsx.demo.logwatch)
– This repository contains a set of applications to demonstrate basic concepts
of SPL and Streams, while working through some real-world examples. The
applications are self-contained, small, easy to understand, with well-defined
problem statements.

Coming Soon…
 Approved Project Proposals
– CDC Toolkit (streamsx.cdc)
• It provides support to efficiently read / write data from CDC.
– Parquet Toolkit (streamsx.parquet)
• Parquet is a columnar storage format for Hadooop. This repository is created for
hosting operators for reading and writing data in Parquet format.
– Location Based Services (streamsx.locationbasedservices)
• It is intended to contain a toolkit that provides generic location based services.
– Logging Toolkit (streamsx.logging)
• It is intended to contain operators and functions for analysing log files.
– Social Toolkit (streamsx.social)
• It is intended to contain operators and functions for integrating Streams with social
media sites.
– Patterns Repository (streamsx.pattern)
• This repository is intended to host pattern classes and common functionality for Java
primitive operators.

Github Releases
 These projects have official releases:
– HDFS, Messaging, HBase and Inet
 Official releases are thoroughly tested and are recommended for
production environment
 Official releases can also be included in the Streams project
 For Inet v2.0 – a subset of operators are included in the product
 Inet v2.5 – contains full set of operators, and should be stable and
work correctly for Streams v4.0.

Github Releases

Support
 Operators / Toolkits that are part of the Streams v4.0 product
– Operators are supported by IBM support channels
– You may report problems by opening PMRs
– You may ask questions on DeveloperWorks
– You may ask questions by opening issues on Github
 Toolkits / Projects that are only on Github
– Projects are supported by committers / contributors on the Github project
– You may ask questions by opening issues on Github – Issues are
monitored.
– You may ask questions on DeveloperWorks
– Committers will provide the best effort to support the project and provide
the help that you need.

How to get Github Toolkits?
 Demo
– Download a release, and add as Streams toolkits in Studio
– Clone a repository and build it at command line or Studio

Enhancement Requests / Report a bug
 Enter enhancement requests and bug reports in the project that
you are working with.
 Vote on existing enhancement / defects!
 For enhancement requests, describe the usecase and how you
intend to use the feature.
 For bug report, will be really helpful if you can describe:
– Version of toolkit used
– SPL snippet
– Problems encountered
– Include domain / PE logs
if possible

Propose a new project
 Process is documented here:
– https://github.com/IBMStreams/administration/blob/master/process.md
 Enter a new issue in the Administration Project:
– https://github.com/IBMStreams/administration
 Proposal should include information on:
– Provide a meaningful title for the issue
– Goal of the project
– Content of initial contribution
– Provide enough information for us to evaluate if the project should be added.
 Project proposal will be evaluated and voted by Managing
Committee.

How to Participate?
 Many new and cool projects to try!
– E.g. AdaptiveParsers, Document
 We want your feedback and input!
– Report any issues you have found
– If you think it’s cool, let us know!
– Which toolkit do you use? How do you use it?
– Do you want the toolkit to be included in the product?
 Contribute code and samples!
– Got a clever way to do things? Contribute it to the samples project!
– Need a new parameter? Need support for a new type of server / data format?
Work with us to try to implement it!
 Got an idea?
– Propose a new project / new feature.

Questions?

Streams GitHub Products Overview for IBM InfoSphere Streams V4.0

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Streams GitHub Products Overview for IBM InfoSphere Streams V4.0

Similar to Streams GitHub Products Overview for IBM InfoSphere Streams V4.0 (20)

More from lisanl

More from lisanl (20)

Recently uploaded

Recently uploaded (20)

Streams GitHub Products Overview for IBM InfoSphere Streams V4.0