Thousands of organizations worldwide already use the ELK stack – or one of its challengers – in order to get real-time insights from their IT assets. When considering WSO2 API Manager, despite its obvious advantages, they can be seemingly unwilling to adopt a new and different statistics oriented product; in other words, WSO2 Data Analytics Server, is not always the option to start with if you have plain statistics needs or if you have invested a lot into some other product. This session explains how to leverage the extensible nature of WSO2 API Manager to collect and analyze API statistics using a third party tool. It gives details on how to write a new data publishing agent, create your own template and configure WSO2 API Manager in order to publish API runtime statistics towards ELK. This talk describes the architecture and the implementation project and gives “how to” guidelines that could be reused to integrate with other products like Splunk or Graylog.

1. WSO2 API Manager sta1s1cs:
integra1on with the ELK stack
Dominique Debailleux
Senior Architect
EMOXA

2. Agenda
• API Management tools: what is it about ?
• API U1liza1on metrics … what for ?
• WSO2 API Manager in a nutshell
• Sta1s1cs: WSO2 DAS approach
• Yet Another Solu1on ?
• Draw me an
• PuJng it all together
• One step beyond
• FAQ
• Almost forgot !
• WSO2 APIM: open source … and open minded

3. API Management tools: what is it about?

4. API Management tooling: what is it about ?
• A li%le reminder, just in case you lived in a cave in the Nunavut during the last 3 years …
• API management (APIM)
– Publishing, promo1ng and overseeing APIs
– In a secure, scalable environment.
– Also includes the crea1on of end user support resources that define and
document the API.
• APIM tools main focus:
– Design & deploy APIs
– Expose them to internal and external developers
– Ensure that they are secured and func1onal
– Get ra1ng & u1liza1on feedbacks

5. API Management tooling: what is it about ?
• APIM tools basics:
– Developer portal: engagement w/partners or developers
– SLA & traffic management
– High availability & scalability
– Consistency across implementa1ons and versions
– Caching mechanism
– Security & authoriza1on
– Easy to deploy and test
– API Documenta1on
– Sta1s1cs & Analy1cs

6. API U1liza1on metrics … what for ?

7. API U1liza1on metrics … what for ?
• Help to diagnos1c how people use APIs and get insights for your business
• Allow organiza1ons
– to understand API u1liza1on
– to make sure the needs of developers and applica1ons using the API are
being met
– to plan accordingly for future enhancements or capaci1es
• Will also be used to help both monitor and mone1ze the APIs exposed

8. API U1liza1on metrics … what for ?
• APIM tools collect and store run1me informa1on about APIs and their
consumers such as:
– how oen are APIs being used (average, peak requests, response 1me, …)
– which APIs are being used,
– who is registrered and is using the APIs,
– SLA compliance or expecta1on,
– …
– well, any data that is meaningfull to the organiza1ons

9. WSO2 API Manager in a nutshell

10. WSO2 API Manager in a nutshell
• h^p://wso2.com/products/api-manager/
• Free and open source APIM solu1on
– Designs & publishes APIs in minutes
– Includes a developer portal w/self registra1on (Applica1on Store)
– Governs APIs (lifecycle, visibility, versions)
– API crea1on and Users registra1on valida1on through workflows (WSO2
BPS)
– Provides security through APIs access control (OAuth2)
– …

11. WSO2 API Manager in a nutshell
• Free and open source APIM solu1on (cont’d)
– …
– On-premise & Saas version
– HA & Scalable
– Limited overhead
– Highly extensible & customizable
– Provides Analy5cs (WSO2 DAS)
– Leverages on WSO2 ESB and WSO2 IS

12. WSO2 API Manager in a nutshell
• Four components which can be split on different servers
• Publisher (management webapp)
– API design and configura1on
– Publica1on toward the Store and the Gateway
• Store (portal webapp)
– APIs catalogue
– Subscrip1ons
– Access key genera1on (Oauth token + Client ID & secret)
– APIs test (sandbox & produc1on)

13. WSO2 API Manager in a nutshell
• Gateway (run1me component)
– API Proxy
– Applies policies (security, thro^ling) and collects sta1s1cs
– Passes the web service call to the backend
– Leverages on WSO2 ESB
• Key Manager (security component)
– Token management
– Authoriza1on management

14. 1
Backend
API Gateway
Extract
access token
Validate
access token
Apply QoS
policies
Publish
sta1s1cs
Custom
extensions
API Store
Key Manager
API Publisher
Client
Publisher
1
1’
2
2
3
4 5
6 7 8
9

15. Sta1s1cs: WSO2 DAS approach

16. 1
Backend
API Gateway
Extract
access token
Validate
access token
Apply QoS
policies
Custom
extensions
Publish
sta1s1cs
Client
Publisher
5
1
2 3
4
API Publisher
API Store
APIM
sta1s1cs
tables
request response
fault
other
event
streams
2
3
2’
3’
2’’
3’’
2’’
3’’
5
Stream
data
table
Summary
data
table
Event flow

17. 1
Backend
APIM
sta1s1cs
tables
BAM mediator
Workflow executor
In
APIMgtUsageHandler
Out
APIMgtResponseHandler
Fault
APIMgtFaultHandler
APIMgtUsageDataBridgeDataPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
API Gateway (ESB inside)
Publish sta1s1cs
Usage publisher
RequestPublisherDTO
Stream data
table
Summary
data table
SPARK
Analy1cs
Event
receivers
Event
store
Processed
event
store

18. APIMgtUsageDataPublisher
+publishEvent (RequestPublisherDTO)
+publishEvent (ResponsePublisherDTO)
+publishEvent (FaultPublisherDTO)
APIMgtUsageDataBridgeDataPublisher
RequestPublisherDTO
Request stream defini=on
ResponsePublisherDTO
Response stream defini=on
FaultPublisherDTO
Fault stream defini=on
APIMgtFaulHandler
+mediate(MessageContext)
APIMgtUsageHandler
+mediate(MessageContext)
APIMgtResponseHandler
+mediate(MessageContext)
DAS Publisher
implementa1on
Handlers &
Mediators
Stream DTOs

19. Yet Another Solu1on ?

20. Yet Another Solu1on ?
• DAS is a great soware but what if one already has a BAM and
SAM solu1on in place ?
• Opera1ves are not eager to add a second (third, …) tool
• Some only need plain sta1s1cs with first level history capabili1es
(summups and simple aggrega1ons) that may not require high
compu1ng analysis model
• … not necessarily -> user story

21. APIMgtUsageDataPublisher
+publishEvent (RequestPublisherDTO)
+publishEvent (ResponsePublisherDTO)
+publishEvent (FaultPublisherDTO)
YourAPIMgtUsagePublisher
RequestPublisherDTO
Request stream defini=on
ResponsePublisherDTO
Response stream defini=on
FaultPublisherDTO
Fault stream defini=on
APIMgtFaulHandler
+mediate(MessageContext)
APIMgtUsageHandler
+handleRequest(MessageCo
ntext)
APIMgtResponseHandler
+mediate(MessageContext)
YOUR Publisher
implementa1on
You keep the same
Stream DTOs

22. 2
Backend
BAM mediator
Workflow executor
In
APIMgtUsageHandler
Out
APIMgtResponseHandler
Fault
APIMgtFaultHandler
YourAPIMgtUsageDataPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
API Gateway (ESB inside)
Publish sta1s1cs
Usage publisher
RequestPublisherDTO
Your preferred
analy1cs solu1on
goes there
With some
adjustments
here

23. Draw me an

24. • ELK
– Pipeline of three open source projects Elas1csearch, Logstash and
Kibana
– Take data from any source, any format
– Search, analyze,
– Visualize it in real 1me
Draw me an

25. • Logstash: 3 stages pipeline
– Inputs - Responsible for collect the data from various sources such
as logs, syslog files, shippers, beats, redis, …
– Filters – makes condi1onal transforma1ons (grok, drop, clone,
1me, geoip, …) like
• parsing – using regular expressions – ,
• adding fields or tags,
• formaJng structures like JSON, …
– Outputs - sends the data to various des1na1ons, like a
Elas1cSearch cluster, files or tools like graphite or nagios
Draw me an

26. • Elas1csearch: RESTful data indexer and search engine
– Solu1on to make searches and analysis on a set of data via a query
language
– Addresses structured , unstrucured data and 1me series
– Distributed, reliable and scalable
– Easy management (elas1csearch-head tools)
– Leverages on Lucene
Draw me an

27. • Kibana: web-based light and easy-to-use dashboard tool
– Queries Elas1csearch indexes and discovers data
– Let you interact with data via graphics (selec1ons, filters, 1me
zooming
– Histograms, pie charts, 1me series and even geo maps created and
populated on the fly
– Visuals that can be combined into dashboards
– Secured access via Shield plugin
Draw me an

28. Draw me an
Beats, tcp,
h^p, udp,
jmx, puppet …
Jdbc, Couchdb,
Sqlite, Redis, ES …
Kaua, Rabbitmq,
Pipe, …
File, Log4j, …
55 output plugins …
50 input plugins …

29. PuJng it all together

30. APIMgtUsageDataPublisher
+publishEvent (RequestPublisherDTO)
+publishEvent (ResponsePublisherDTO)
+publishEvent (FaultPublisherDTO)
APIMgtUsageLogPublisher
+publishEvent (RequestPublisherDTO) {
log.info(data from RequestPublisherDTO);
}
…
RequestPublisherDTO
Request stream defini=on
ResponsePublisherDTO
Response stream defini=on
FaultPublisherDTO
Fault stream defini=on
APIMgtFaulHandler
+mediate(MessageContext)
APIMgtUsageHandler
+handleRequest(MessageCo
ntext)
APIMgtResponseHandler
+mediate(MessageContext)
LOG Publisher
implementa1on

31. 3
Backend
BAM mediator
Workflow executor
In
APIMgtUsageHandler
Out
APIMgtResponseHandler
Fault
APIMgtFaultHandler
APIMgtUsageLogPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
API Gateway (ESB inside)
Publish sta1s1cs
Usage publisher
RequestPublisherDTO
stats
log
files

32. PuJng it all together
Logstash output
w/sta1s1cs

33. PuJng it all together

34. PuJng it all together

35. PuJng it all together

36. PuJng it all together

37. PuJng it all together
• Easy uh ?
– One class to create
– One configura1on line to modify
– Some fun with
– That’s it
• Let’s try something more challenging

38. One step beyond

39. One step beyond
• What if I want to enrich the provided sta1s1cs with my own
ones ?
– No correla1on between request and response
– No space for you beloved data
• Well, let’s restart from the class diagram

40. APIMgtUsageExtendedLogPublisher
+publishEvent (ExtendedRequestPublisherDTO)
+publishEvent (ExtendedResponsePublisherDTO)
+publishEvent (ExtendedFaultPublisherDTO)
ExtendedRequestPublisherD
TO
Request stream defini=on
ExtendedResponsePublisher
DTO
Response stream defini=on
ExtendedFaultPublisherDTO
Fault stream defini=on
APIMgtFaulLogHandler
+mediate(MessageContext)
APIMgtUsageLogHandler
+handleRequest(MessageCo
ntext)
APIMgtResponseLogHandler
+mediate(MessageContext)
Extended Log
Publisher
RequestPublisherDTO
Request stream defini=on
ResponsePublisherDTO
Response stream defini=on
FaultPublisherDTO
Fault stream defini=on
Extra fields to log
• Correla1on Id
• Business Data
+ get extra
values from the
mc
Exis1ng sta1s1cs
data

41. 4
Backend
BAM mediator
Workflow executor
In
APIMgtUsageLogHandler
Out
APIMgtResponseLogHandler
Fault
APIMgtFaultLogHandler
APIMgtUsageExtendedLogPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
API Gateway (ESB inside)
Publish sta1s1cs
Usage publisher
ExtendedRequestPublisherDTO
WSO2
APIM
log
files
API media1on
is impacted

42. One step beyond Request
Response
Logstash output
w/business data and
correla1on id
Logstash output
w/sta1s1cs

43. One step beyond

44. FAQ

45. FAQ
• What if I want to populate the WSO2 APIM sta1s1cs tables
anyway ?
– DAS approach,
– DIY approach
• What if I don't (want to) use the ELK stack ?
• What about the 1.x versions ?
• What if I republish the API (“one step beyond” case) ?
• What about the overhead ?

46. Almost forgot !

47. Almost forgot
• WSO2 Preferred Partner
• WSO2 Training France
• Using WSO2 solu1ons Since 2013
• Barcelona WSO2 Con 2014 speaker
• SOA Architects
co-authors of the book
“SOA – a guide for architects”

48. WSO2 APIM: open source … and open minded

49. WSO2 APIM: open source … and open minded
• WSO2 APIM is the right tool to manage your APIs.
• Most API Tools have to connect to an exis1ng ecosystem in place
• WSO2 APIM has proven its capabili1es to embrace real life cases
and to adapt.
• Extension capabili1es of WSO2 APIM is unchallenged and "part
of its strength”
• An “elas1c” solu1on … in a way

50. Thank You!
#WSO2ConEU
Share your feedback for this session
wso2con.com/app