Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Extending WSO2 Analytics Platform

507 views

Published on

WSO2 provides a complete platform for data analytics through the WSO2 analytics platform. It revolutionizes the way you work with and understand your data. By uniquely combining simultaneous batch, real-time, interactive, and predictive analytics, you can turn data from Internet of Things (IoT), mobile and Web apps into actionable insights. The WSO2 analytics platform comes with a rich set of features that support the required analytics needs and has the additional capability of being flexible and extensible.

In this webinar, we will

Introduce the WSO2 analytics platform
Examine extensions including
Real-time analytics (Siddhi extension)
Batch processing extensions
Predictive analytics extensions
EventReceiver and EventPublisher extensions
Outline the benefits of WSO2’s analytics platform through real-world customer case studies

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Extending WSO2 Analytics Platform

  1. 1. Webinar: Extending WSO2 Analytics Platform Mohanadarshan Vivekanandalingam Associate Technical Lead
  2. 2. Agenda ● Introduction to WSO2 analytics platform ● Examine extensions including ○ Real-time analytics (Siddhi extension) ○ Batch analytics extensions ○ Event Receiver and Event Publisher extensions ○ Predictive analytics extensions ● Outline the benefits of WSO2’s analytics platform through real-world customer use cases. 2
  3. 3. WSO2 Data Analytics Server 3
  4. 4. WSO2 Data Analytics Server 4 • Fully-open source solution with the ability to build systems and applications that collect and analyze both realtime and persisted data and communicate the results. • High performance data capture framework • Highly available and scalable by design • Pre-built Data Agents for WSO2 products
  5. 5. Real-time Analytics 5
  6. 6. Realtime Analytics Extensions 6 ●This includes Siddhi Extensions ■ Custom Function ■ Custom Window ■ Custom Aggregate ■ Custom Stream Function ■ Custom Stream Processor
  7. 7. Function Extension 7 ● Consumes zero or more parameters for each event and output a single attribute as an output. ● This could be used to manipulate event attributes to generate new attribute like Function operator. ● Extend org.wso2.siddhi.core.executor.function.FunctionExecutor from InValueStream select math:sin(inValue) as sinValue insert into OutMediationStream;
  8. 8. Window Extension 8 ● Allows events to be collected and expired without altering the event format based on the given input parameters like the Window operator. ● Default Window types - Length, Time, Unique and etc.. ● Extend org.wso2.siddhi.core.query.processor.stream.window.WindowProcessor from TempStream#window.custom:customWindow(10) select * insert into AvgRoomTempStream ;
  9. 9. Aggregate Extension 9 ● Consumes zero or more parameters for each event and output a single attribute (having an aggregated results based in the input parameters as an output). ● Used with conjunction with a window in order to find the aggregated results based on the given window. ● Default Aggregators - sum, max, avg and etc.. ● Extend org.wso2.siddhi.core.query.selector.attribute.aggregator.AttributeAggregator from pizzaOrder#window.length(20) select custom:count(orderNo) as totalOrders insert into orderCount;
  10. 10. Stream Function Extension 10 ● Allows events to be altered by adding one or more attributes to it. (Simply, can output multiple outputs) ● Events can be output upon each event arrival ● Extend org.wso2.siddhi.core.query.processor.stream.function.StreamFunctionProcessor from geocodeStream#geo:geocode(location) select latitude, longitude, formattedAddress insert into dataOut;
  11. 11. Stream Processor Extension 11 ● Allows to alter an event format ● Considered as Window++ ● Extend org.wso2.siddhi.core.query.processor.stream.StreamProcessor from baseballData#timeseries:regress(2, 10000, 0.95, salary, rbi, walks, strikeouts, errors) select * insert into regResults;
  12. 12. Batch Analytics Extension 12 •User Defined Functions (UDF) •Aggregators for Lucene Indexing •DataSource Connectors (Eg: HBase, Cassandra & etc..)
  13. 13. User Defined Functions (UDF) 13 ● Apache Spark allows UDFs (User Defined Functions) to be created if you want want to use a feature that is not available for Spark by default. public class StringConcatonator implements CarbonUDF { /** This UDF returns the concatenation of two strings */ public String concat(String firstString, String secondString) { return firstString + secondString; } } • Add below to DAS_HOME/repository/conf/analytics/spark/spark-udf-config.xml <udf-configuration> <custom-udf-classes> <class-name>org.wso2.customUDFs. StringConcatonator</class-name> ... </custom-udf-classes> </udf-configuration>
  14. 14. Aggregators for Lucene Indexing 14 WSO2 DAS contains 5 default Lucene based aggregated functions. ● MIN ● MAX ● SUM ● AVG ● COUNT Users can add custom aggregator function for Lucene by extending below interface. org.wso2.carbon.analytics.dataservice.core.indexing.aggregates.AggregateFunction (DAS 3.1.0 onwards) Refer mail thread - [Architecture] [Analytics] Improvements to Lucene based Aggregate functions (Installing Aggregates as OSGI components)
  15. 15. Datasource Connectors 15 DAS supports below datasource connectors by default. ● RDBMS ● Cassandra ● HBASE ● HDFS Extension can be written by implementing the below interface, org.wso2.carbon.analytics.datasource.core.rs.AnalyticsRecordStore https://docs.wso2.com/display/DAS310/Configuring+Data+Persistence
  16. 16. Predictive Analytics 16
  17. 17. Predictive Analytics Extensions 17 •Dataset Processors •Input Adapters •Model Builders •Output Adapters
  18. 18. Input Adapters 18 ● Used to read data from different storages such as files, HDFs and registry. ● Can create an ML Input Adapter by implementing the MLInputAdapter interface.
  19. 19. Dataset Processors 19 ● Each data source should have an implementation of DatasetProcessor. ● ML supports File, HDFS and DAS as data sources. Therefore we have the following implementation classes.
  20. 20. Model Builders 20 ● ML model generation can be extended by implementing MLModelBuilders. ● Currently we have a supervised spark model builder and an unsupervised spark model builder. ● If you need to extend model generation to some other library or a new algorithm type, you can use this extension point of WSO2 ML.
  21. 21. Output Adapters 21 ● Used to write data to different storages such as files, HDFS and registry. ● Can create an ML Output Adapter by implementing the MLOutputAdapter interface.
  22. 22. Event Receiver Extensions 22 ● Allows to receive events from different data sources.. ● Implemented with OSGI whiteboard pattern.
  23. 23. Event Publisher Extensions 23 •Allows to push events to various data sinks. •Implemented with OSGI whiteboard pattern.
  24. 24. Case Studies from Real Customers 24
  25. 25. Pacific Controls Pacific Controls is an innovative company delivering an IoT platform of platforms: Galaxy 2021. The platform allows to manage all kinds of devices within a building and take automated decisions such as moving an elevator or starting the air conditioning based on certain conditions. Within Galaxy2021, CEP is used for monitoring alarms and specific conditions.Pacific Controls also uses other products from the WSO2 platform, such as WSO2 ESB and Identity Server. https://www.youtube.com/watch?v=OG0N7cfaJ_8
  26. 26. UBER http://www.infoq.com/presentations/uber-stream-processing UBER uses WSO2 CEP to detect fraud. P.S : Does not pay for us (Opensource at work ! ).
  27. 27. 27 A leading Airlines uses CEP to enhance customer experience by calculating the average time to reach their boarding gate (going through security, walking, etc.). They also want to track the time it takes to clean a plane, in order to better streamline the boarding process and notify both the airline and customers about potential delays. They evaluated WSO2 CEP first as they were already using our platform and decided to use it as it addressed all their requirements. The Cleveland Clinic, ranked among the top 3 hospitals in the US, uses a Clinical Intelligence Platform that combines big data storage, stream and batch processing to provide decision support to clinicians. Real-time analytics for the platform is provided by WSO2 CEP along with custom extensions to handle healthcare data.
  28. 28. Few more use cases 28
  29. 29. 29 US Election Monitor https://wso2.com/election2016/
  30. 30. SUPER BOWL 50 - BigData Game http://wso2.com/landing/big-data-game/
  31. 31. 31 Fraud Detection 31 • Use or change the generic rules we provide and add as many rules as they like • Change weights of Fraud Scoring Model to suit their business needs • Use the Markov Modelling and Clustering capabilities to learn unknown Fraud Patterns in their domain • Use the dashboard provided or plug the Fraud Detection Toolkit to their own Fraud Detection UI http://wso2.com/library/webinars/2015/02/catch-them-in-the-act- fraud-detection-with-wso2-cep-and-wso2-bam/ https://www.youtube.com/watch?v=aLwG4thHOXg
  32. 32. ESB Analytics ESB Analytics can be used to collect statistics, debug, and profile your mediation sequences. https://docs.wso2.com/display/ESB500/ESB+Analytics
  33. 33. Conclusion ● Next WSO2 Analytics Platform release contains many bug fixes, improvements and features. ○ Incremental Processing - Batch Analytics ○ Siddhi Performance Improvements - Realtime Analytics ○ Siddhi Debugger ○ Analytics features for ESB, APIM, IS, IOT and etc.. ○ Cross Tenant Data Retrieval in Super Tenant Spark Queries ○ Custom Lucene Aggregators ● Stay tuned for next release and related updates. 33
  34. 34. CONTACT US !

×