Splunk at Shazam.

1. Copyright © 2016 Splunk Inc.
SplunkZam!
Chris Kammermann
Service Engineering Infrastructure (Team Lead)

2. 2
Warning!
This presentation may involve Audience Participation!

3. 3
• During SplunkLive!, you will be able to Shazam the following:
• Audio or Bluetooth
• 09:45 - 10:04 Shazam to find out more about me
• 10:05 - 10:30 Fill out a survey on the Shazam presentation
• 10:30 - 13:00 Splunk Conf
• 15:30 - 17:00 Splunk Survey Monkey
• Images – All Day:
• Splunk .conf in Orlando
• Shazam For Brands
Shazamability Schedule

4. 4
Agenda
• Me as represented by a Splunk dashboard
• Shazam, the Audio/Visual/Beacon recognition app
• How Shazam used Splunk in the beginning
• The rebirth of Splunk, how Shazam is using it now and what is planned
for the future

5. 5
Index=personal sourcetype=anecdote novelty>0

6. 6

7. 7
Name that Song Sight, Sound
• One of the worlds most loved and downloaded
Mobile Apps
• 120 million Monthly Active Users
• Shazaming leads to 400K music downloads every day!
5% of all music downloads originate via Shazam!
• Shazam can recognise TV Adverts, Advertising
Billboards, Coke Cans (USA), KFC buckets (Australia),
Print Magazines (esquire), even some TFL London
Buses!

8. 8
Sometime 4 Years Ago
• Legacy relational databases and a mass of unstructured
data caused the following challenges for Shazam:
• Inability to gain insight into the app or our customers
• Lengthy data processing times
• Struggled to react quickly
• We began using Splunk to address these problems

9. 9
Beacon Data
• Every Handset, Wearable Device or PC that has Shazam installed sends
Beacon data to servers in the cloud
• Almost every button click will generate a beacon log
• Hundreds of Gigs per day of Beacon data is ingested into Splunk. Most
events are searchable in Splunk < 4 seconds from the button click!
• At the time, Beacon data was 80+% of the data stored in Splunk
• Few OS logs were being sent to Splunk

10. 10
How Shazam used this data
• Shazam for TV campaign analysis
• A/B testing
• Music Charts.
• Shazam top 20 Radio Show (Australia)
• Mobile App error analysis
• Key Monthly Reports ie: MAU

11. 11
TV Advert Analytics Dashboard

12. 12
A/B(C/D) Testing
• New features in the App could could
grow our MAU. Worst case is that these
features annoy the users and they
uninstall the app, never to return again!
• Every fraction of a percent of our user
base that is happy = potentially tens of
thousands of users every month!

13. 13
Ad Hoc or Targeted Queries
• How many devices in Japan have Bluetooth enabled?
• How many people in Los Angeles like this band?
• What songs have an artificially inflated tag count? Is someone or
something trying to rig the charts?
• What song is popular 8th Avenue and 14th Street New York?
• How many people share their song find on Google Plus?

14. 14
Good Enough for a Splunk Case Study!
“Splunk enables us to analyze all of our mobile app
data without having to do batch processing or any
other cost and time-intensive steps required of
traditional business intelligence. Now we can
change metrics or add new dashboards quickly and
easily, and provide the latest results to our
partners and internal stakeholders in real time.
That just wasn’t possible before.”
Charles Henrich, EVP Engineering Shazam

15. 15
Problems?
• Very first iteration, Architected sub optimally for performance
• Not enough storage. The business wanted *ALL* the data for *ALL*
time stored and searchable in Splunk!
• Hundreds of reports and dashboards created. Started to become
bogged down. Our monthly active user report would take >1 week to
run!
• Splunk admins moved onto other projects and the system was left to
‘run quietly in the corner’
• Often exceeded the license limit

16. 16
We solved this by
• Assigning the right people to support the platform
• Scaled out and clustered our Indexer nodes
• Used ‘expired’ hardware. 600 unused (and out of warranty)
servers that we could choose from. It shouldn’t matter if a node
fails as we can quickly re-provision and use Puppet to
reconfigure. Savings of >80% when compared to buying
new/supported servers.
• We shrunk the data coming in using the sedcmd facility
• We bought a bigger license!

17. 17
ALL the Data for ALL the time!
• But we still couldn’t store ALL the data for ALL the time in Splunk.
• So we ended up accessing Long Term data that we stored in Amazon
Red Shift using Splunk DB Connect.

18. 18
Rebirth!
• July 2015
• Version 6.22!
• Bigger! Faster! Better!
• OS and other system logs now being ingested.
• ‘DevOps’ insight screens being developed
• Ingesting more and more logs from other systems as users
get excited by the possibilities

19. 19
New Uses: Heart Rate Monitor
Sparklines show co-
related Nagios alerts!
In this situation we
were able to identify
a service the flooded
the network switch
once an hour which
caused a common
switch to drop
packets

20. 20
New Uses: What happened overnight?

21. 21

22. 22

23. 23

24. 24
New Uses: MAU and Capacity Predictions
A new release caused our 4 hourly tagging rate in a strategic developing market to jump.
Splunk ‘predict’ command shows the predicted impact of this change

25. 25
Can Splunk predict the next #1 hit?
• Fun Question to ask!
• One of our non Splunk systems can predict up to 33 days out the
number one hit on the Billboard charts. There is a great video
which explains this phenomenon
https://www.youtube.com/watch?v=mcTPvxo8SXY
• So far we have been unsuccessful in using Splunk Predictive
Analytics on the music charts

26. 26
New Use Case: Inventory
(Q) How do we get a single view of hardware we are charged for? Our data
sources are:
- our internal inventory database - accessed via REST API
- once a month excel spreadsheet from our external Data Center provider
- Amazon AWS
(A) Splunk! With the following Components:
• Splunk AWS App
• REST API Modular Input
• inputcsv command!

27. 27
New Use: Animated Map???
Analysis of Top 10 Music Tracks over a 24 hour period

28. 28
New Use: Animated Map???
Analysis of Top 10 Music Tracks over a 24 hour period

29. 29
Future Use: DevOps, Anomaly Detection
• Key objective is to release better code, quicker. Integrating, Git, JIRA,
Jenkins, Puppet, Virtualisation and Container logs into Splunk.
• We have known unknowns and sometimes unknown unknowns.
Anomaly detection should help us identify these?!

30. THANK YOU
Use Shazam to fill out a survey on this talk