Enterprise systems such as enterprise resources planning (EPR), system monitoring, and stock transaction and financial transaction systems generate an enormous amount of events that can contain useful data; processing these data efficiently can help to gain a competitive advantage. Implementing custom solutions to capture, analyze, and act on these Big Data imposes challenges and operational complexities. WSO2 CEP and WSO2 BAM together provide a solution to deliver a low-latency, high-volume, and scalable environment enabling data collection, real-time as well as batch analysis and firing notifications of multiple types across numerous endpoints.
10. Data Collection through Adaptors
● Can receive events via
SOAP, HTTP, JMS, ..
● WSO2 Events is highly
optimized version
(400K events TPS)
● Default Agents and
you can write custom
agents.
Agent agent = new Agent(agentConfiguration);
publisher = new AsyncDataPublisher(
"tcp://localhost:7612", .. );
StreamDefinition definition =
new StreamDefinition(STREAM_NAME,
VERSION);
definition.addPayloadData("sid", STRING);
...
publisher.addStreamDefinition(definition);
...
Event event = new Event();
event.setPayloadData(eventData);
publisher.publish(STREAM_NAME, VERSION, event);
37. Outline
● Introduction to Business Activity Monitoring
● WSO2 Business Activity Monitor
● WSO2 BAM Components
● WSO2 BAM Toolboxes
● WSO2 BAM High Availability Distributed Deployment
● Demo
38. Business Activity Monitoring
“The aggregation, analysis, and presentation of real-time
information about activities inside organizations and
involving customers and partners” - Gartner
42. WSO2 Business Activity Monitor
● Fully-open source solution for aggregating and analyzing data
and presenting information about business activities.
● Cloud-enabled, lightweight, developer-friendly and
easy-to-deploy
● High performance data capture framework
● Elastically-scalable data analysis powered by Apache Hadoop
● Pre-built Data Agents for WSO2 products
45. Data Agents
● Compatible with CEP/BAM
● Get data across to BAM
− Service monitoring feature – WSO2 AS, DSS, ESB, API Manager
− Mediation monitoring feature – BAM Mediator for WSO2 ESB
− Custom data-agents
● Asynchronous & non-blocking
● Thrift for high performance message throughput
46. Custom Agents
● Data agents are meant to be installed/plugged into the server
to be monitored
● Through Thrift, different languages can be supported
48. Data Receiver
● Receives data and stores it in Cassandra
○ Scalable, big data repository
● Asynchronous & non-blocking
○ Combination of Cassandra, Thrift and the non-blocking nature results in
extremely fast writes
● Shared with WSO2 CEP for real time analysis
● Supports Thrift & REST API
49. Data Model
● Data is sent over using strongly typed Data Streams
{
'name':'phone.retail.shop',
'version':'1.0.0',
'nickName': 'Phone_Retail_Shop',
'description': 'Phone Sales',
'metaData':[
{'name':'clientType','type':'STRING'}
],
'payloadData':[
{'name':'brand','type':'STRING'},
{'name':'quantity','type':'INT'},
{'name':'total','type':'INT'},
{'name':'user','type':'STRING'}
]
}
● Data Streams are versioned
○ Allows for easier analysis
51. The Analyzer Engine
● Powered by Apache Hadoop with querying/managing through
Apache Hive
● Data transfer to and from data sources is handled through
custom storage handlers
● Parallel, distributed processing through the MapReduce
programming model
52. The Analyzer Engine
● Runs on local Hadoop node or delegates to Hadoop cluster
○ Scalable analytics
○ Cluster can range from a couple of nodes to 1000s
● Analysis is carried out based on analytics scripts
● Scripts are based on an easy-to-learn, SQL-like query language
INSERT OVERWRITE TABLE UserTable SELECT userName, COUNT(DISTINCT
orderID),SUM(quantity) FROM PhoneSalesTable WHERE version= "1.0.0" GROUP
BY userName;
53. The Analyzer Engine
● Scripts can be scheduled
○ Ex: once a minute, every Wednesday at 4:15 p.m., every 30th
at 12 midnight
● Polyglot data architectures are supported
○ Write summarised information to any kind of datastore
○ Custom written Hive JDBC handler used by default for output to relational DBs
55. The Presentation Layer
● Gadget Portal for the dashboard
● Gadget Generation Wizard – Tool for generating custom
gadgets for the dashboard
● Activity Dashboard for correlating activities
● Message Console
● Plug in your own report server/ dashboard server
58. BAM Toolboxes
● BAM Toolboxes are installable and hot deployable artefacts
used for deploying functionalities to a BAM server
○ Stream definitions
○ Analytics scripts
○ Dashboards
● Supports plugging in of one or more of the above features
● Toolboxes for monitoring and auditing most WSO2 products are
available OOTB
● Toolboxes for custom scenarios can be created easily
59. HA Distributed BAM deployment
● WSO2 BAM can be clustered and deployed in a distributed
manner to enable high-availability, fail-over scenarios
○ Partially distributed deployment
■ Cassandra (storage) and Hadoop (analyzer) nodes are clustered
○ Fully distributed deployment
■ All components of BAM are clustered (data receiver, storage, analyzer and
presentation)
● Hazelcast in-memory data grids are used for clustering
implementation