This document provides an overview of Bloom filters, including what they are, why you would want to use one, practical examples of their use, and how they work. Bloom filters are a data structure that can be used to test if an element is present in a set. They have a tiny memory footprint compared to storing the full set, use constant memory, and allow extremely fast lookups. However, they can produce false positives. The document discusses examples of Bloom filters being used by companies like Akamai, Medium, and Cassandra to reduce storage usage and improve performance. It also explains how Bloom filters work under the hood using hash functions and bit arrays.

1. Hayden Marchant, Software Engineer
haydenm@wix.com
Bloom Filters

2. Agenda
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

3. A
Bloom-what?
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

4. Just a Data Structure
‘Is an element in a set
or not’

5. Probabilistic

6. Definitely not in set
Probably in set
OR

7. ILLUSTRATION

8. Why would I
want one?
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

9. Why not just
store all used
keys?

10. Element
Key size:
# elements:
Memory
required:
100 bytes
10 million
100 * 10M
= 1GB
SIMPLE MATHS

11. Bloom Filters:
Tiny footprint
~ 10 bits per element

12. >98%
reduction
in space
Element
Key size:
# elements:
Memory
required:
10 bits
10 million
100M bits
~= 12MB
SIMPLE MATHS
(PART 2)

13. Footprints:
Traditional = 1GB
Bloom Filter <=12MB

14. Practical
Examples
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

15. Akamai
Prevent caching of
One-Hit-Wonders
75% of web requests are
only ever called once
- Called One-Hit-Wonders
Use Bloom Filters to prevent
one-hit-wonders to be stored
in Disk Cache
1. Saves disk-access for
these 75% of requests
2. reducing workload

16. Medium
Avoid recommending
articles a user has
read
- Add article recommended
to bloom filter -
$userid-$articleid
- Before recommending,
check in Bloom Filter if
(user,article) exists.
- If not, then recommend
and then add to bloom
filter

17. Cassandra
Reduce disk lookup for
non-existent rows in file
storage
- Shard responsible over
range of data
- Each shard stored in
immutable shard-files
- Create Bloom Filter in
server for each shard-file
- Do not access shard-files
if Bloom Filter returns
false for key

18. Chrome
Identifying
Malicious URLs
Store malicious
URLs in Bloom
Filter
In-depth check on
positive response

19. How it works
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

20. Basics of
Bloom
Filter
● Array of m bits
● initially set to 0
● Hash functions
● k hash functions
defined
● maps or hashes some
set element to one of
the m array positions

21. Bloom Filter
in
slow motion

22. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
{ }m = 20
k = 3

23. 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0
{ cat}m = 20
k = 3

24. 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 0 1
{ cat , dog }m = 20
k = 3

25. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1
{cat , dog , mouse }m = 20
k = 3

26. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1
{cat , dog , mouse }
owl
m = 20
k = 3
owl is definitely NOT in set

27. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1
rat
m = 20
k = 3
rat might be in set
{cat , dog , mouse }

28. Constant memory
space

29. Extremely Fast
Processing

30. Low Error Rate

31. The maths
‘Under-the-hood’

32. 1
2
3
4
5
PROBABILITIES
101
Probability of slot not selected for a single hash function
Probability of slot not selected for all k hashes
Probability of slot not selected for all k hashes of n elements
Probability of slot selected for all k hashes of n elements
Probability of k slots selected for n elements === False positive

33. GETTING A LITTLE
TRICKIER

34. If you want to read
more on the maths,
take a look at
https://en.wikipedia.org/
wiki/Bloom_filter

35. A Simple
Calculator
Click here: https://hur.st/bloomfilter

36. Next step
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

37. Bloom
Filters in
Code
● Libraries exist in most
languages
○ https://github.com/alexan
drnikitin/bloom-filter-scala
○ https://www.npmjs.com/p
ackage/bloom-filters
● Distributed Bloom Filters
in Redis
○ Perfect for multiple
instance of a service
using single bloom filter

38. Now, go think how using
Bloom Filters in your
services could reduce
payload and speed things
up

39. Other
probabilistic
structures
● Counting Bloom Filters
○ Allow deletes
● TopK
○ Keep track of Top
K counts of
elements
● Count-Min Sketch
○ Frequency table of
elements

40. Q&A