This document outlines a highly scalable feed aggregator powered by HBase and Python. It describes the table structure using three tables to store feeds, extracted URL data, and an index. Source code is available on GitHub along with detailed install instructions. Key lessons learned are that HBase does not support relations, joins, or sophisticated queries so these must be handled in application code, and that indexes should be designed for time sorting and character encodings converted to bytes.

1. feed aggregator powered by
hbase & python
Andrei Savu
wurbe #25

2. Objectives
Highly scalable feed aggregator
Play with python & thrift
Provide some sample code
Provide detailed install instructions
Learn new stuff

3. Table Structure
3 tables: Feeds, Urls, UrlsIndex
Feeds: all feeds
Urls: data extracted from feeds
UrlsIndex: index table

4. Source code
http://github.com/andreisavu/feedaggregator
detailed install instructions

5. Lessons learned

6. Lesson #1: Hbase Game Rules
Not relations
No joins
No sophisticated query engine
No column typing
No transactions
No secondary indices
... all done in application code

7. Lesson #2: Design your index
<cat>/<w3c_timestamp>
time sorting = lexicographic sorting

8. Lesson #3: No charsets
convert everything to bytes
... but store the original charset

9. Questions?
http://www.andreisavu.ro
http://twitter.com/andreisavu
contact@andreisavu.ro