What Shazam doesn't want you to know

What Shazam doesn't want you to know
Roy van Rijn Software Craftsman

Imagine this...

Friday afternoon

Getting inspiration

Singing along...

Meet Stewie!

Music matching

Shazam is magic... alien technology!
Start the application.
Let it listen for +/- 20 seconds.
It tells you:
Music matching

Saturday morning

Specifying the audio format:
private AudioFormat getFormat() { float sampleRate = 44100; int sampleSizeInBits = 8; int channels = 1; boolean signed = true ; boolean bigEndian = true ; return new AudioFormat( sampleRate, sampleSizeInBits, channels, signed, bigEndian); }
The microphone

Accessing the microphone
final AudioFormat format = getFormat(); DataLine.Info info = new DataLine.Info(TargetDataLine. class , format); final TargetDataLine line = (TargetDataLine) AudioSystem. getLine (info); line.open(format); line.start();
The microphone

Reading the sound:
out = new ByteArrayOutputStream(); running = true ; try { while ( running ) { int count = line.read( buffer , 0, buffer . length ); if (count > 0) { out .write( buffer , 0, count); } } out .close(); } catch (IOException e) { throw new RuntimeException(e); }
The microphone

0 0 0 1 2 5 0 -1 -3 -4 -5 -2 0 1 2 0 2 (etc)
Inside the byte array

Plotting a graph

Membrane, cochlear, brain
The human ear

Hertz
The amount of cycles per second
i.e. one sine wave.
Frequencies...?

To match music we need frequencies, not waves.
Time vs Frequency

Fourier Transformation

Excellent explanation by Stuart Riffle
http://altdevblogaday.com
Fourier Transformation

We've lost track of time!
Frequency domain

Solution: Apply transformation on pieces
byte audio[] = out .toByteArray(); final int amountSlices = audio. length / SLICE_SIZE ; Complex[][] results = new Complex[amountChucks][]; for ( int slice = 0;slice < amountSlices; slice++) { Complex[] complex = new Complex[ SLICE_SIZE ]; for ( int i = 0;i< SLICE_SIZE ;i++) { complex[i] = new Complex(audio[(slice* SLICE_SIZE )+i], 0); } results[slice] = FFT. fft (complex); }
Windowing

From wikipedia:
Spectum Analyzer A spectrum analyzer or spectral analyzer is a device used to examine the spectral composition of some electrical, acoustic , or optical waveform.
Spectrum Analyzer

Demo: Aphex Twin

Sunday

Determine the key points (in ranges):
34 41 92 129 186
39 41 117 130 218
40 42 106 129 191
40 47 117 121 217
40 53 81 129 208
40 48 109 132 260
39 45 89 135 247
40 42 84 125 251
40 41 81 121 232
38 42 113 131 245
(etc...)
Matching the song

Playing/decoding MP3 files:
JLayer (real time MP3 decoder)
jl1.0.1.jar
MP3SPI (Java plugin, based on JLayer)
mp3spi1.9.4.jar
Tritonus (implementation of Java Sound API)
tritonus_share.jar
Something to match against

Harvesting my music collection:
public void harvest(File rootDirectory) { String[] itemsInDirectory = rootDirectory.list(); for (String itemInDirectory:itemsInDirectory) { if (itemInDirectory.endsWith( ".mp3" )) { //Assume mp3 file File mp3File = new File(mp3Directory, itemInDirectory); captureAudio(mp3File); } else if ( new File(mp3Directory, itemInDirectory).isDirectory()) { //Directory? Recurse! harvest( new File(mp3Directory, itemInDirectory)); } } }
Something to match against

We have:
Set of +/- 3000 files of reference data (songs)
Way of capturing key moments with microphone
Lets do some matching!
What we have now

Create a single hash per slice
private static final in t FUZ_FACTOR = 2; private long hash(String line) { String[] p = line.split( "t" ); long p1 = Long. parseLong (p[0]); long p2 = Long. parseLong (p[1]); long p3 = Long. parseLong (p[2]); long p4 = Long. parseLong (p[3]); // long p5 = Long.parseLong(p[5]); // Not using the fifth point currently return (p4-(p4% FUZ_FACTOR )) * 100000000 + (p3-(p3% FUZ_FACTOR )) * 100000 + (p2-(p2% FUZ_FACTOR )) * 100 + (p1-(p1% FUZ_FACTOR )); }
Hash function

Load all the reference hashes
Listen to the microphone and generate hashes
Find all matching hashes
Return the reference-song with most hits
This worked (a bit) but produced a lot of mis-hits
How can we improve this?
Matching algorithm #1

Microphone, sample #1, matches with:
Song 4, sample #4
Song 6, sample #9
Song 4, sample #6
Song 6, sample #10
Song 8, sample #4
Song 4, sample #5
Song 5, sample #3

Song 4, sample #4: 4 – 1 = 3
Song 6, sample #9 9 – 1 = 8
Song 4, sample #6 6 – 2 = 4
Song 6, sample #10 10 – 2 = 8
Song 8, sample #4 4 – 2 = 2
Song 4, sample #5 5 – 3 = 2
Song 5, sample #2 2 – 3 = -1

Now we group the results:
2x: Song 6 with offset 8 1x: Song 4 with offset 2 1x: Song 4 with offset 3 1x: Song 4 with offset 4 1x: Song 8 with offset 2 1x: Song 5 with offset -1 Matching algorithm #2

Music matching (Shazam, SoundHound) isn't magic
Searching can be done very fast:
Searching for hashes: O(1)
Align and match
The Fourier Transformation is like a spirograph:
Recap

What are other uses for this algorithm:
Speech recognition?
Probably not..
Detecting duplicate songs in your music collection?
Yes! Took 5 minutes for crude implementation
Subtitle synchronisation in India !
Other uses for this algorithm

... Landmark Digital Services owns the patents that cover the algorithm used as the basis for your recently posted “Creating Shazam In Java”. While it is not Landmark’s intention to alienate those in the Open Source and Music Information Retrieval community, Landmark must request that you do not ship, deploy or post the code presented in your post. Landmark also requests that in the future you do not ship, deploy or post any portions or versions of this code in its current state or in any modified state. We hope you understand our position and that we would be legally remiss not to make this request. We appreciate your immediate attention and response. ... Landmark Digital Services

After this email I contacted:
Arnoud Engelfriet (Dutch IT lawyer, patent attorney)
Free Software Foundation
And others.
Getting information

From another email:
As I'm sure you are aware, your blogpost may be viewed internationally. As a result, you may contribute to someone infringing our patents in any part of the world. While we trust your good intentions, yes, we would like you to refrain from releasing the code at all and to remove the blogpost explaining the algorithm .
Now the blogpost?

My reply was short and concise:
I'm sorry, I can't comply. The blogpost will absolutely not be removed. Good luck.
No way...

Questions?

There are a couple I haven't tried:
Non-linear time scales
Better 'important'-point selection (not static ranges)
HTML 5 recording, processing with Javascript
Any improvements?

What Shazam doesn't want you to know

by Roy van Rijn on Nov 18, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 2

Statistics

What Shazam doesn’t want you to know — Presentation Transcript