Facebook Spotlight is the hack that we built in the Facebook Mobile Hack Hong Kong 2012. The idea is to crawl facebook's data, index it and provide full text search. All these happen in javascript and made possible by HTML5 powerful features.

1. Search my social circle…
STANLEY . F STANLEY . N KEUNG . H STEVE . N
Facebook Mobile Hack Hong Kong - 22nd March 2012

2. Current facebook search bar can only search people, pages, events, etc
page 2

3. But can we search by user profile details?
page 3

4. Can we search by the comments?
page 4

5. Can we search by the event details?
page 5

6. Build a search engine for my social circle!
ONLY in JAVASCRIPT!

7. Our Idea
• Crawl data from my social circle provided by Facebook Graph API
• Build inverted index from the content
page 7

8. Why Javascript?
 We love Javascript!
 To demonstrate how powerful Javascript can
be
 For a facebook application, it should be more
ethical to process users’ data in client side
page 8

9. Benefits of Client Side Indexing
 Computation power and storage is now off-
loaded to client side browsers
 facebook does not need to manage the huge
amount of index data
 The search result is very personalized!
You are searching for your own data and that
is what you are interested in!
page 9

10. Demo

12. Auto-complete

13. Query: “baby” – search all objects containing “baby”

14. We are searching facebook
objects based on the content!

15. Query: “birthday @photos” – search all photos object containing “birthday”

16. Query: “single @profile” – search all profile object containing “single”

17. Challenges
 Indexing performance
 Persistent storage for index
 Implementing inverted index algorithm
page 17

18. Web Workers Web DB
page 18

19. Indexing Performance
 For each user, we do a batch API call to
Graph API already
 However, it still takes on average 3.8
seconds for the data to return
 Most of my friends has around 300 friends
 So it takes 19 minutes to complete the
crawling!?!?!?
page 19

20. Web Workers come to save me!
Time to index 100 facebook users
400
350
300
Time (seconds)
250
200
150
100
50
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Number of web workers
page 20

21. Web Workers
Web Workers,
forked by main thread
Crawl data and
build sub-index
Main thread
merge all sub-indices
page 21

22. Web Workers
 Web workers has much limitation, e.g. you
cannot access the DOM
 You also cannot import the facebook SDK or
jQuery library. It DOES NOT WORK!
 Luckily, you can still use the most
fundamental function - xmlhttprequest
 So we have to use xmlhttprequest to
simulate the FB.api function in batch mode
page 22

23. Web DB
 Web DB provides us a solution for permanent
offline storage
 My friends list has 556 users
 Generating a search index having ~18,000
English keywords
 Only takes less than a second to save and
load from the Web DB
page 23

24. Problems still need to tackle
 Asian language (Chinese, Japanese, etc)
tokenization
 Search relevancy improvement
 Need a more standardized query language
page 24

25. Inverted Indexing concept
page 25

26. Conclusion
 Client side full text search is made possible
with HTML5’s web workers and storage!
 How to put it forward?
 What if we are not using facebook’s data?
 Index content of viewed websites by users?
 Index RSS feeds?
 How to play with the indexed data?
page 26

27. Credits
 Javscript full text search engine from
TheLadders.com
 iQuery Masonry
 Backbone.js
page 27

28. Stanley Fok
Stanley Ng
Steve Ng
Hon
Happy hacking!