Presentation by Brandwatch Software Developer Dr Tim Owen at the second Big Data Brighton meetup, hosted by Brandwatch: www.brandwatch.com

1. Scaling the Brandwatch search index – a history
Tim Owen: tim@brandwatch.com | Tel: +44 (0)1273 234 290

2. Overview: Brandwatch Social Media Monitoring
• We crawl, index and store as much of the useful web as we can
• Customers define search queries to find pages of interest
• The page content and metadata is stored in
• Traditional relational databases – PostgreSQL
• Full-text search index – Solr/Lucene
• Key-value stores – Voldemort and HBase
• Our real-time queries across this data are typically
• How many pages match my search query, per day, over the last 3 months?
• What are: the top websites? the most prolific authors? the sentiment mix?
• What topics are being discussed among these mentions of my search query
© 2012 Brandwatch | www.brandwatch.com 2

3. Big Data
•
Total data set is (reasonably?) Big Data – 100TB+
•
But we don't currently use Hadoop/Map-Reduce
•
Why not? It doesn't fit our use-cases. Map-Reduce is...
•
Batch-orientated, not real-time
•
Runs in minutes/hours/days, not milliseconds
•
Suited to scans across all or most of a dataset
•
We do real-time queries for small subsets of the data
•
Indexes are essential
•
Relational DBs like Postgres are good at this, but scaling is tricky
•
HBase is good for this use-case, and scales horizontally
© 2012 Brandwatch | www.brandwatch.com 3

4. Search Index
•
Our full-text search index has grown to 25TB in 5 years
•
The growth is definitely not linear!
•
We have changed the architecture of it several times
•
Solving the pain point each time opened up new ones
© 2012 Brandwatch | www.brandwatch.com 4

5. Search Mk 1 (monolithic)
1 Crawler
•
2007 to 2008
•
Up to 80GB
•
Separates reads and writes
1 Lucene Index
•
Reads are from local disk
Rsync nightly •
Have redundancy
Index Index
•
Stale data – 1 day old
Replica 1 Replica 2 •
Monolithic index
Reader Reader
Client 1 Client 2
© 2012 Brandwatch | www.brandwatch.com 5

6. Search Mk 2 (shared filesystem)
2-3 Crawlers
• 2009
• Up to 350GB
NFS • Real-time
• Reads and Writes mixed
1 Lucene Index • No redundancy
on Shared F/S
• NFS reads are slow
• Clients open index in RAM
NFS
• Monolithic – Doesn't Scale
Reader Reader
Client 1 Client 2
© 2012 Brandwatch | www.brandwatch.com 6

7. Search Mk 3 (partitioned)
3-20 Crawlers
• 2010 to mid 2011
• Up to 8.5TB
NFS • Only read partitions needed
• Only write to 1 partition
Jun Jul Aug
• Easier to scale
Sep Oct Latest
• Still reading over NFS
• Still opening index in RAM
NFS
Reader Reader
Client 1 Client 2
© 2012 Brandwatch | www.brandwatch.com 7

8. Search Mk 4 (Solr)
20-30 Crawlers
• 2011 to now
• Up to 25TB
Http
• Offloads RAM from clients
Solr Writer
• Scales horizontally
replication
Solr1 Solr2 Solr3
• Partitions spread over Solrs
• More expensive in H/W
Solr4 ... Solr24
• Failover is not automatic
Http
• No auto-balancing partitions
Reader Reader
Client 1 Client 2
© 2012 Brandwatch | www.brandwatch.com 8

9. Search Mk 5 (SolrCloud)
30+ Crawlers
• 2013
• 25TB+
• Handles failover
• Manages partition placing
SolrCloud • New tech – early days
• ?? always something..
Reader Reader
...
Client 1 Client 2
© 2012 Brandwatch | www.brandwatch.com 9

10. CONTACT
EMAIL: contact@brandwatch.com
WEB: http://www.brandwatch.com
TWITTER: @brandwatch
PHONE:
UK: +44 (0)1273 234 290
US: +1 212 229 2240
Germany: +49 (0)711 912 442 04
FAX:
UK: +44 (0)1273 234 291
DOCUMENT LIMITATION
The information given in this document has been checked for accuracy and completeness however Brandwatch
shall not be liable for any errors or omissions.
Brandwatch is a trading name of Runtime Collective Limited. Registered in England & Wales: 3898053
4th Floor, International House, Queens Road, Brighton, BN1 3XE, United Kingdom
© 2012 Brandwatch | www.brandwatch.com 10