Understanding Business APIs through statistics


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Understanding Business APIs through statistics

  1. 1. Understanding Business APIs through Statistics Amila De Silva Dinusha Senanayaka
  2. 2. About WSO2 • Providing the only complete open source componentized cloud platform – Dedicated to removing all the stumbling blocks to enterprise agility – Enabling you to focus on business logic and business value • Recognized by leading analyst firms as visionaries and leaders – Gartner cites WSO2 as visionaries in all 3 categories of application infrastructure – Forrester places WSO2 in top 2 for API Management • Global corporation with offices in USA, UK & Sri Lanka – 200+ employees and growing • Business model of selling comprehensive support & maintenance for our products
  3. 3. 150+ globally positioned support customers
  4. 4. Overview ● Role of Statistics in API Management ○ Statistics to understand user communities ○ As tool for making strategic decisions. ● Different Options for analysing data ○ Phases in Data analysis ○ Offline vs Real-time analysis ● BAM as an offline data analytic engine ○ How large event streams gets summarized ● CEP for real time processing ○ Architecture of CEP ○ Stages of an execution plan ● GA as a feedback channel for app developers ● Options for visualising data. ○ Generating reports out of BAM summarized data.
  5. 5. APIs - the way to go ● APIs are no longer a luxury - they are becoming a necessity. ● API - A common interface for disparate platforms ● More open you are more users you will attract. ● Build an ecosystem around your APIs.
  6. 6. Knowing your users
  7. 7. Why do you need statistics? ● There are many connected components ● One wrong move can make disastrous consequences. ● Feedback channels are important - ○ See how users access them and learn - It explains what they say. ● Discover hidden potentials within your communities ○ Going for a new billing model - See how users access them. ● Can you check them all manually?
  8. 8. What we provide ● Two branches in analysis ○ Offline Data analysis ○ Realtime Analaysis ● Offline Data Analysis ○ Gives you a general idea of your APIs ○ Who your users are. ○ Leads you to make long term decisions. ● Realtime analysis ○ Identifies co-occurrence of several useful business flows. ○ Find and act on it.
  9. 9. How we analyse it ● Done in two main phases ○ Data Gathering ○ Analysis. ● Gather data at product level. ○ multiple event streams for request, response and faults ○ each invocation as an event ● Delegating analysis to a different product ○ Simply any tool capable of processing event streams can analyse data. ○ Currently BAM and CEP are used. ● Extensions to achieve product specific analysis ○ In BAM this is done through a hive script ○ In CEP this can be done by a transport adaptor + execution plan.
  10. 10. Data Gathering ● Event = Invocation ● In the gateway we collect data related to each invocation ● Three different event streams ○ Request Stream ○ Response Stream ○ Fault stream
  11. 11. Structure of a Stream Request Stream 'payloadData':[ {'name':'consumerKey','type':'STRING'}, {'name':'context','type':'STRING'}, {'name':'api_version','type':'STRING'}, {'name':'api','type':'STRING'}, {'name':'resource','type':'STRING'}, {'name':'method','type':'STRING'}, {'name':'version','type':'STRING'}, {'name':'request','type':'INT'}, {'name':'requestTime','type':'LONG'}, {'name':'userId','type':'STRING'}, {'name':'tenantDomain','type':'STRING'}, {'name':'hostName','type':'STRING'}, {'name':'apiPublisher','type':'STRING'}, {'name':'applicationName','type':'STRING'}, {'name':'applicationId','type':'STRING'} ] Response Stream 'payloadData':[ {'name':'consumerKey','type':'STRING'}, {'name':'context','type':'STRING'}, {'name':'api_version','type':'STRING'}, {'name':'api','type':'STRING'}, {'name':'resource','type':'STRING'}, {'name':'method','type':'STRING'}, {'name':'version','type':'STRING'}, {'name':'response','type':'INT'}, {'name':'responseTime','type':'LONG'}, {'name':'serviceTime','type':'LONG'}, {'name':'userId','type':'STRING'}, {'name':'tenantDomain','type':'STRING'}, {'name':'hostName','type':'STRING'}, {'name':'apiPublisher','type':'STRING'}, {'name':'applicationName','type':'STRING'}, {'name':'applicationId','type':'STRING'} ] Fault Stream 'payloadData':[ … {'name':errorCode','type': 'STRING '}, {'name':‘errorMessage','type': 'STRING '}, .. ]
  12. 12. Analysing Data with BAM...
  13. 13. Analysing Data with BAM ● Used for offline Analysis. ● Aggregates, stores, then analyses data. ● Capable of receiving events from multiple data agents. ○ In APIM 1.5.0 we use a load balancing data publisher. ● All the incoming events are first written to a Cassandra DB ● Data analysed using a hive script running on top of a hadoop cluster. ● Summarised data written back to a Relational DB.
  14. 14. Analysing Data with BAM... Summarized tables
  15. 15. Analysing Data with BAM... Table Descriptions
  16. 16. Analysing Data with BAM... Visualizing summarized data
  17. 17. Analysing with BAM... Summarized data usages for billing ● Decide on a billing model ● Integrate with existing billing engines
  18. 18. Real-time Analysis with CEP ● APIM can publish events to multiple data agents. ● Same streams are used to publish events to CEP ● Whenever it receives a new event CEP executes a query and evaluates the event. ● Siddhi queries define the evaluation criteria.
  19. 19. CEP Architecture
  20. 20. CEP Architecture... ● Transport Adaptors to read/write event sources. ○ We can create alternate event streams out of the original streams. ● Event builders convert events to a standard format. ● Queries defined as query plans. ● Event processor executes a query plan. ○ It might generate additional events. ● Event formatters convert back the output.
  21. 21. Example ● Task : Identify client applications sending too many repeated requests ○ Too Many : 5 requests ○ repeated : within 1 minute ● How to identify a client ○ Consumer key And the query is: from AMRequest#window.time(1 min) insert into outStream count(consumerKey) as myCount,api_version,api,consumerKey group by consumerKey,api_version,api having (myCount > 5);
  22. 22. Flow in CEP... ● Transport adaptor reads requestStreamDefn ● Creates a second input stream AMRequest ○ This only contains subset of data. ● Event builder transform the input stream ● Query is defined in a query plan. ○ Even processor executes each of the registered query plans. ● Upon meeting the condition an event is generated. ● Event formatter converts the event. ● Mail is sent through the transport adaptor.
  23. 23. Monitoring Statistics with Google Analytics • API Manager can be easily configured with Google Analytics <GoogleAnalyticsTracking> <Enabled>false</Enabled> <TrackingID>UA-XXXXXXXX-X</TrackingID> </GoogleAnalyticsTracking>
  24. 24. Monitoring Statistics with Google Analytics
  25. 25. Visualising ● Needed for interpreting statistics. ● Gadgets provided in BAM ● Plugin any external tool for generating reports ● Combining summary tables to create new views.
  26. 26. Summary • Statistics as a channel to verify business objectives. • WSO2 API Manager provides different mechanisms to analyze data. • WSO2 BAM can be integrated with API Manager process offline data. • Gather -> Cassandra • Analyze -> Relational DB • WSO2 CEP can be integrated with API Manager and real time data analyzing. • Google Analytics integration with API Manager.
  27. 27. Engage with WSO2 • Helping you get the most out of your deployments • From project evaluation and inception to development and going into production, WSO2 is your partner in ensuring 100% project success