This document describes a real-time Twitter search engine that allows searching tweets based on keywords, country, language, and negative words. It utilizes ElasticSearch's percolator to register user queries and match incoming tweets in real-time. Logstash is used to build a scalable high-throughput backend data pipeline. SocketIO keeps an open connection between the front-end client and Flask server for real-time updates. Challenges addressed include designing the percolator data pipeline and decoupling the backend database from the frontend.

1. SEARCH
YOUR
TWEETS
SEARCH
LIKE
A
PROFESSIONAL

2. Motivation
• Twitter
represents
a
rich
flow
of
information
• Lack
of
an
effective
way
to
query
the
twitter
• Hard
to
monitor
interested
topics
at
real
time

3. Search
Tweets
Like
a
Professional
A
Real
Time
Twitter
Search
Engine
That
Allows
you
to
Search
based
on:
•Keywords
◦Country
◦Language
◦Negative
words
Demo(http://searchyourtweet.info:5000/input)

4. Keep
an
eye
on
your
interested
topic
•Express
your
interest,
we
will
keep
you
update
on
the
newest
event
•Video
(https://youtu.be/GdRmXNfukos)

5. Data
pipeline
Query
Controller
Backend
Database
percolator
Logic
Layer Frontend
Searching
database
Data
Backup
Pub/Sub
Publish
Matching
query
Register
query
searching

6. Real
Time
Monitor
on
Twitter
◦Implemented
using
ElasticSearch Percolator
◦Think
it
as
“search
in
reverse”
◦ User
register
queries
into
percolator
◦ Percolator
match
incoming
documents
with
registered
queries
◦Challenge:
◦ How
to
design
the
percolator
data
pipeline?
◦ How
to
decouple
the
backend
database
with
frontend
server?
◦ Use
publish
/
subscribe
design
pattern

7. Real
Time
Monitor
Data
Flow
Percolator
Query
database
Twitter
database
Controller
Pub/Sub
subscribe
Open
channel

8. Challenge
Build
a
high
throughput
real
time
backend
data
pipeline?
• Use
Logstash!
◦ Highly Scalable
◦ Compatiblewith
different
sources
and
destination
A
scalable
high
throughput
pipelineCurrent
backend
pipeline

9. Challenge
• Real
time
update
on
frontend
client:
• Instead
of
using
“setInterval()”
javascript function,
I
use
“socketIO”
to
keep
socket
open
between
front-­‐end
client
and
flask
server
• Construct
ElasticSearch query
• Use
python
requests
library
to
query
ElasticSearch
• Fine
tuning
on
ElasticSearch

10. About
Me
M.Math,
University
of
Waterloo
◦ Field:
Statistics
and
Machine
Learning
B.S.,
University
of
Toronto
◦ Field:
Applied
Mathematics
Data
Scientist
Intern,
Neon
Inc.,
San
Francisco
Back-­‐end
Model
Developer,
MetricAid Inc.,
Toronto
Experience
in
Deep
Learning:
◦ Convolutional
Network,
Recurrent
Network
•OS/161
(a
simplified
POSIX
OS)

11. Questions?
Thank
you!

12. Parallelization
of
percolator
• Will
consumes
a
lot
hardware:
O(mn)
• Another
choice:
Luwak +
Samza