Steve Halverson is a developer with the IBM Streams platform team. In this presentation, Steve covers the details of how IBM Streams V4.1 integrates with IBM Platform Symphony. Please view the related presentation available at: http://www.slideshare.net/lisanl/introduction-to-ibm-platform-symphony-integration-with-ibm-streams-v41

1. © 2015 IBM Corporation
IBM Streams and Platform Symphony
IBM Streams Version 4.1
Steve Halverson
shalver@us.ibm.com

2. 2 © 2015 IBM Corporation
Important Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL
PURPOSES ONLY.
WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE
INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY
OF ANY KIND, EXPRESS OR IMPLIED.
IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY,
WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.
IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR
OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR
THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT
GOVERNING THE USE OF IBM SOFTWARE.
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion. Information regarding potential
future products is intended to outline our general product direction and it should not
be relied on in making a purchasing decision. The information mentioned regarding
potential future products is not a commitment, promise, or legal obligation to deliver
any material, code or functionality. Information about potential future products may
not be incorporated into any contract. The development, release, and timing of any
future features or functionality described for our products remains at our sole
discretion.
THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.
IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

3. 3 © 2015 IBM Corporation
Platform Symphony
 Known for efficient distribution of system resources
 Shipped with Big Insights
 Allows administrator to group resources and grant permissions to allocate
them
 Provides ability to borrow and lend resources dynamically

4. 4 © 2015 IBM Corporation
Platform Symphony Resource Groups
 Ability to group hosts together as a resource pool
 Resource plans indicate how resources are distributed and what type of
borrow/lending policies are to be enforced
 Define “slots” in terms of cores, cpus, etc
– Each slot is a “resource” in streams terminology
 Consumers are granted permissions to allocated slots from the resource
group
 Use resource groups as a basis for our resource tag definitions
 Resource groups are configured in Platform Symphony Web Console

5. 5 © 2015 IBM Corporation
Platform Symphony Consumers
 Has an “execution user” for starting containers
– Can specify root as the execution user
– security.runAsRoot
– instance.runAsUser
– instance.canSetPeOSCapabilities
 Need permission to allocate resources from resource groups
 Map domain/instance to an associated consumer for running streams
services
 Consumers are configured in Platform Symphony Web Console

6. 6 © 2015 IBM Corporation
Distribute and install resource package
 If NOT using a shared file system, a resource package must be
installed on each Platform Symphony host.
 mkresourcepkg [-d,--domain-id <did>] [--pkg-dir <pkg-dir>] [--install-dir
<installation-directory>] [--owner <owner>] [--group <group>] [--minimal] [-h,--
help] [--trace <level>] [-v,--verbose [<level>]]
 rsdeploy to distribute to hosts
 rsdeploy is a Platform Symphony application
 source /opt/ibm/platformsymphony/profile.platform
 <package name> can be something like streams4.1
 rsdeploy add <package name> -p <path to resource package> -u <user> -x
<password>
 rsdeploy install <package name> -c /InfoSphereStreams/streamsroot - r
<resource group> -u <user> -x <password>
 rsdeploy uninstall <package name> -c /InfoSphereStreams/streamsroot - r
<resource group> -u <user> -x <password>
 Do NOT uninstall while domain is up and running
 rsdeploy status <package name> -u < user> -x <password>

7. 7 © 2015 IBM Corporation
Configuration Files
 Files located in
<install>/InfoSphere_Streams/4.1.0.0/etc/symphony
 streams-symphony.properties
 Configuration properties for resource manager and symphony connection
 USER=<admin user to symphony>
 PASSWORD=<admin password to symphony>
 CONSUMER_PROPERTIES=$STREAMS_INSTALL/etc/symphony/consumer.properties
 LOG_PATH=/tmp/streams-on-symphony_$USER
 LOG_LEVEL=error
 DEFAULT_RESOURCE_GROUPS=<group>
 Resource group to use if not specified by tag definition
 RESOURCE_INSTALL_PATH=$STREAMS_INSTALL
 For shared file system; use same install path as resource manager
 RESOURCE_INSTALL_PATH=<install-path>
 For non-shared file system; installation on remote host
 EGO_MASTER_LIST=<hostname>
 Space separated host names for symphony server
 EGO_KD_PORT=<port>
 Non-SSL port for symphony server
 consumer.properties
 Mapping of streams domain/instance to symphony consumer

8. 8 © 2015 IBM Corporation
Consumer Properties
 Map streams domain/instance to symphony consumer
 Evaluated in precedence order
 Syntax
 <default>=consumer
 Consumer to use if does NOT match any other entry
 <management>=consumer
 Consumer to use for management resource
 domain/<management>=consumer
 Consumer to use for domain management resources if no specific
instance entry
 domain=consumer
 Consumer to use if no specific instance entry
 domain/instance/<management>=consumer
 Consumer to use for management resources for specific instance
 domain/myinstance=myconsumer
 Consumer to use for specific instance

9. 9 © 2015 IBM Corporation
Evaluation order for domain resource
1) domain/<management>=consumer (mgmt resource)
2) domain=consumer
3) <management>=consumer (mgmt resource>
4) <default>=consumer

10. 10 © 2015 IBM Corporation
Evaluation order for instance resource
1) domain/instance/<management>=consumer (mgmt resource)
2) domain/instance=consumer
3) domain/<management>=consumer (mgmt resource)
4) domain=consumer
5) <management>=consumer (mgmt resource>
6) <default>=consumer

11. 11 © 2015 IBM Corporation
Configuring Streams Domain
 mkdomain --property domain.externalResourceManager=symphony
 Indicates that domain will be using the Platform Symphony Resource
Manager (i.e. streams-on-symphony)
 setdomainproperty domain.externalResourceManager=symphony
 Convert a domain to use Platform Symphony Resource Manager
 domain.managementResourceAllocationTags=authentication,audit,jmx,sws
 Indicates that all management services will be but on the same
resource
 Modify to: authentication,audit
 aas and auditlog on same resource
 jmx on an additional resource
 sws on an additional resource
 instance.managementResourceAllocationTags=management,view
 Indicates that all management services will be on the same resource
 Modify to: view
 sam,sch,srm on same resource
 view on an additional resource

12. 12 © 2015 IBM Corporation
Configuring resource tag definitions
 Resource tag definition indicates which resource group(s) to use
 mktag --property resourceGroups=<csv resource group names> application
 settagproperty --tag application resourceGroups=<csv resource group
names>
 DEFAULT_RESOURCE_GROUP from properties file if no definition.

13. 13 © 2015 IBM Corporation
Using streams hosts along with resource manager
 Traditional streams hosts can be used in conjunction with Platform
Symphony resources
 Use streams hosts first
 Request any additional resources form Platform Symphony
 Streams hosts must be tagged properly
Example 1:
domain.managementResourceAllocationTags=authentication,audit,jmx,sws
myhost1 - authentication,audit,jmx,sws
Example 2:
domain.managementResourceAllocationTags=authentication,audit
myhost1 - authentication,audit,jmx,sws
myhost2 - authentication,audit
myhost3 - jmx,sws

14. 14 © 2015 IBM Corporation
streams-on-symphony
<install>/bin/streams-on-symphony
Usage: streams-on-symphony command <options>
COMMAND OPTIONS
status Displays status of the Streams Resource Manager.
[--zkconnect {<host>:<port>},...]
[-d,--domain-id <optional: domain identifier> | --all]
start Starts the Streams Resource Manager.
[--zkconnect {<host>:<port>},...]
[-d,--domain-id <optional: domain identifier>]
[--properties <optional: path to configuration properties>]
[--host <optional: host name or ip to use for registration]
[--port <optional: port to use for registration]
[--fg]
stop Stops the Streams Resource Manager.
[--zkconnect {<host>:<port>},...]
[-d,--domain-id <optional: domain identifier>]
[--hosts <optional: comma delimited host names>]
[--force]

15. 15 © 2015 IBM Corporation
streams-on-symphony start
 Start resource manager for multiple domains
 Do not need any other options
 Registers with zookeeper indicating it can accommodate multiple
domains
 Start resource manager for a specific domain
 -d <domainId>
 Registers with zookeeper indicating it is specific to a single domain
 Specify configuration properties file
 --properties <filename>
 May want to secure symphony credentials, etc
 Otherwise uses <install>/etc/symphony/streams-symphony.properties

16. 16 © 2015 IBM Corporation
streams-on-symphony stop
 Stop resource manager(s) for multiple domains
 Do not need any other options
 Stop resource manager(s) for a specific domain
 -d <domainId>
 If -d was specified on start; should also be specified on stop
 Stop resource manager(s) on specific host(s)
 Use --hosts <comma delimited hosts>

17. 17 © 2015 IBM Corporation
streams-on-symphony status
 Display resource manager(s) for multiple domains
 Do not need any other options
 Display resource manager(s) for a specific domain
 -d <domainId>
 Display all resource managers
 --all
 Displays resource managers registered for multiple domains and any
resource manager that is for a specific domain

18. 18 © 2015 IBM Corporation
High Availability
 Running streams-on-symphony multiple times creates multiple
resource managers
 Leader has RUNNING status
 Standby has WAITING status
 If leader terminates; standby takes over
[shalver@g0601b06 bin]$ streams-on-symphony status --all
Host Port PID Status Domain
myhost10.ibm.com (10.1.100.10) 46297 26436 RUNNING <multiple>
myhost11.ibm.com (10.1.100.11) 43189 26590 WAITING <multiple>
myhost12.ibm.com (10.1.100.12) 53235 26639 WAITING <multiple>
myhost13.ibm.com (10.1.100.13) 43630 26777 RUNNING mydomain
myhost14.ibm.com (10.1.100.14) 40975 26832 WAITING mydomain

19. 19 © 2015 IBM Corporation
Demo
1) mkdomain
2) startdomain
3) settagproperty
4) mkinstance
5) startinstance

20. 20 © 2015 IBM Corporation
Questions?