This document discusses using Splunk to analyze logs from Akamai Cloud Monitor in real-time. Akamai Cloud Monitor delivers logs within 60 seconds, enabling real-time operational monitoring and analytics. The document outlines how logs are configured and delivered in JSON format to a Splunk receiver. It also describes building a Splunk application to gain insights into availability, performance, security and release monitoring using the enriched Akamai log data.

1. Copyright © 2015 Splunk Inc.
Leela Kesireddy
Performance Management
PayPal, Inc.
Splunk for
Akamai Cloud Monitor

2. 2
Agenda
2
• Why we need Log Analysis
• Real Time logging OpIon
• Benefits
• ConfiguraIon Details
• Akamai Splunk ApplicaIon

3. 3
Why we need log analysis
3
• Customer interacIons on Akamai exceed 1 BN/day
• It is business criIcal that we :
– Know the state of the site
– Track the impact of changes
– Analyze the customer experience
• Akamai standard log collecIon takes hours to get logs
into NetStorage.
• Downloading logs hourly and indexing them in Splunk
makes them available for analysis.
• But this is not a real Ime soluIon.
DC #1 DC #2
WAF
Log
Gatherin
g
Akamai
Network

4. 4
Real Time Logging
4
• Akamai Cloud Monitor delivers logs in < 60 sec
• Faster delivery enables real Ime operaIonal
monitoring and easier analyIcs by internal users.
• OperaIonal goals required Real Ime visibility into
site & Improved usability of logs.
• Receiving the logs in <60 sec. makes them
usable for real Ime analysis.
– The NOC can immediately see the effects of site
changes.
– The CDN team can see the effects of Akamai
configuraIon changes in real Ime
DC #1 DC #2
WAF
Aggregation
Real Time
Delivery
CM Log
Streaming
Akamai
Network

5. 5
Benefits – Log usability
5
• Akamai CM logs use JSON forma^ng.
• Users can work with the JSON forma`ed
logs more intuiIvely.
• CM contains more rich informaIon like
Geo, Network Performance and WAF
data.
• OpIonal data fields enable rich analyIcs
• Data from the connecIon, HTTP header
or payload can be inserted into CM as
custom data.
• Splunk TA for CM logs put them into CIM
format for ES consumpIon

6. ConfiguraIon Details

7. 7
CM Data Delivery
7
• Select Data Sets to Include from Default &
OpIonal Data Sets
" Default: CP, Format, Message, Type, Version
" OpIonal: Akadebug, network, netPerf, Geo, WAF,
PPCustomData
• Configure the AggregaIon opIons:
– Time: 60 seconds max
– Line Count: Max 3000 records
– Message Size: Max 900 KB of data
• Configure Delivery Endpoint, DistribuIon &
Failover OpIons:
– Primary receiver gets 100%
– Secondary receiver gets 100% if Primary
receiver is unavailable
– NetStorage gets 100% if Primary &
Secondary are unavailable
ê With scheduled FTP download hourly

8. 8
CM Receiver
8
• Receiver Build
– Akamai Posts CM data to receiver VIP
– SSL cert required for secure connecIon by Akamai
– MulIple Linux servers in pool running nginx & node.js
– Writes logs to a local data file
– Splunk Universal Forwarder monitors logs and forwards them to Indexers

9. 9
CM Receiver (Future State)
9
CM Receiver (Future State)
9
Splunk
VIP
CM VIP1
CM VIP2
CM VIP1
CM VIP2
• Plan for receiver redundancy
in the iniIal design
NetStorage

10. 10
New- Splunk 6.3 HTTP Event Collector
" HTTP endpoint can securely receive high-
volume JSON-based applicaIon and IOT
data.
" Create and mange receiver configuraIons
using the HTTP event collector
configuraIon.
" Token based authenIcaIon Model
" Supports to both h`p & h`ps
" Replaces the nginx & node.js receiver soluIon
10

11. Akamai ApplicaIon

12. 12
Splunk ConfiguraIon
12
• Index Configured with 100 day retenIon
• Built Add-on’s with Field renames & EvaluaIons &
normalize data to be CIM Complaint.
• App was built to gain insights into
– Real Time Traffic
– Insights by Property
– Monitor Health & Performance of Origin
– Monitor Heath & Performance of Akamai Edge

13. 13 13
• Performance Issue
– Searches Were Slow
– Dashboards taking longer to load
– Greater than 25K events/sec
• Summary Index
– Loses the Rich informaIon in data
• Report AcceleraIon
– AcceleraIon is suspended as the summary reaches 10% of its total size
– No Control on Summary Schedule
Hurdles
Data Models & AcceleraIon
Came to Rescue

14. 14
Availability & Performance
14
Availability
• Volume of Hits.
• Success & Failure Volume by
HTTP Status.
• Success & Failure by GEO
Performance
• Track total Ime for transacIons.
• Origin Latency
• Last Mile Round Trip Time

15. 15
Origin & Edge
15
Origin
• Monitor traffic distribuIon.
• Error Monitoring.
• Monitor Latency.
• Edge Errors
• Edge Performance stats
• Last Mile RTT
Edge

16. 16
Security
16
• Monitor the WAF denies and warnings.
• Monitor Top Deny Rules.
• Report on the WAF warning triggered.
• Top deny & warning URL’s.
• Deny & warn tracking by Geo
• Top Denied Client IP’s

17. 17
Release Monitoring
17
• Monitor Traffic by Origin / DC
• Monitor Issue’s & Performance by
Property
• Performance by Origin
• Last Mile Time by Edge
• Performance by GEO
DC 01 DC 02 DC 03 DC 04 DC 05
DC 01 DC 02 DC 03 DC 04 DC 05

18. Final Analysis
Q & A

19. 19
Final Analysis
19
• The iniIal business needs for Real Time logging have been met:
– Real Ime monitoring enables us to know the current state of the site
– Dashboards allow us to track the effect of site changes
– JSON data forma^ng makes it easier to do analyIcs on the customer experience
• The addiIonal benefits add significant value that is not possible to get any
another way:
– Dashboards for NOC and support teams to see current state of the site
– Header data used for troubleshooIng site and code issues
– Header informaIon providing customer experience analyIcs data
– Performance data on a regional and network level
• Download the Splunk App for Akamai CloudMonitor
– h`ps://splunkbase.splunk.com/app/2923/

20. Thank you