demo1

1. Improvement for
StackOverflow.com
Chentao Zhang
Insight Data Engineering SV

2. Motivations

3. Motivations
1.Tag right?
2. When will be answered?
~Help users to tag their questions on
stackoverflow.com more properly
DEMO:
www.stackoverflowtags.tech

4. Data Pipeline
Historical
data(60G)
Streaming data

5. Input Data
Question:
{
“post_id”:67172,
“post_date”:”6-10-2015-00-01-02”,
“type”:0,
“parent_id”:0
“tiltle”:” Java Exception”,
“body”:”....”,
“tags”:“java;algorithm”,
“user_id”:782,
…
}
Answer:
{
“post_id”:67172,
“post_date”:”6-10-2015-00-01-23”,
“type”:1,
“parent_id”:67172
“tiltle”:” “,
“body”:”You should....”,
“tags”:“”,
“user_id”:1982,
…
}

6. Query #1
• Prob. of a question labeled with specific tag(such as tag A) will be
answered in 10 mins
= number of questions answered in 10 mins and tagged with A/ total number of questions tagged with A

7. question_id tags answer_time(sec) posted_at
231 Java 3010 2016_01_02_21_20_01
290 spark 7381 2016_01_02_22_09_01
341 Java 5611 2016_01_10_01_02_05
Query #1
• Prob. of a question labeled with specific tag(such as tag A) will be
answered in 10 mins
= number of questions answered in 10 mins and tagged with A/ total number of questions tagged with A

8. question_id tags answer_time(sec) posted_at
231 Java 3010 2016_01_02_21_20_01
290 spark 7381 2016_01_02_22_09_01
341 Java 5611 2016_01_10_01_02_05
Query #1
• Prob. of a question labeled with specific tag(such as tag A) will be
answered in 10 mins
= number of questions answered in 10 mins and tagged with A/ total number of questions tagged with A

9. Shard
Shard Shard
Shard
Request
Query #1

10. Shard
Shard Shard
Shard
0.5|2000
0.3|2000
0.1|1660
(Prob.|sample size)
0.2|2000
Request
Query #1

11. Shard
Shard Shard
Shard
0.5|2000
0.3|2000
0.1|1660
0.2|2000
1660*0.1+2000*0.2+2000*0.5+2000*0.3)/
(2000*3+1660)=0.28
Request
Query #1

12. Query #2
• Recommend a list of tags which are similar to a
specific tag and sorted by their similarity to the tag

13. spark/tags_graph:
{
“id”:261
”tags”:[”kdc:lat”,
“java”,
“hadoop:java”,
“apache”,
“linux:OS”]
}
*vertice:tags
*weight of vertice: number of people who have answered questions labeled with this tags
*weight of edge: number of people who have answered questions for both tags
* Similarity of A to B = WAB /WA
Query #2

14. • User entered tagA
Query #2
tag A
tag C tag B
tag D
3
2
5
5 10
3
7

15. • User entered tagA
• Search all neighbors
of tagA and compute
their similarity to A.
Query #2
tag A
tag C tag B
tag D
3
2
5
5 10
3
7
SC->A=3/5=0.6
SD->A=2/3=0.67
SB>A=5/10=0.5

16. • User entered tagA
• Search all neighbors
of tagA and compute
their similarity to A.
• Sort B,C,D by their
similarity to A
Query #2
tag A
tag C tag B
tag D
3
2
5
5 10
3
7
SC->A=3/5=0.6
SD->A=2/3=0.67
SB>A=5/10=0.5

17. • User entered tagA
• Search all neighbors of
tagA and compute their
similarity to A.
• Sort B,C,D by their
similarity to A
• Give result to user
Query #2
tag A
tag C tag B
tag D
3
2
5
5 10
3
7
SC->A=3/5=0.6
SD->A=2/3=0.67
SB>A=5/10=0.5

18. Challenges and Future Considerations
• Streaming processing to update information
• Process big data
• Scale up the performance of sorting in graph

19. About Me
• Chentao(Sam) Zhang
• MS in Electrical & Computer
Engineering from University of
Delaware
• Passionated to learn and try
new things

20. Query #1
tags
Prob. of being answered in
10 mins Avg time(sec)
Java 0.32 1200
spark 0.013 31000

21. tags
Prob. of being answered in
10 mins Avg time(sec)
Java 0.32 1200
spark 0.013 31000
Query #1

22. Challenges
~Process big data
~Performance of updating batch of
data

23. tags
Prob. of being answered in
10 mins Avg time(sec)
Java 0.32 1200
spark 0.013 31000
question_id tags answer_time(sec) posted_at
231 Java 3010 2016_01_02_21_20_01
290 spark 7381 2016_01_02_22_09_01
341 Java 5611 2016_01_10_01_02_05
Query #1

24. tags
Prob. of being answered in
10 mins Avg time(sec)
Java 0.32 1200
spark 0.013 31000
question_id tags answer_time(sec) posted_at
231 Java 3010 2016_01_02_21_20_01
290 spark 7381 2016_01_02_22_09_01
341 Java 5611 2016_01_10_01_02_05
Query #1

25. Data Modelling
~Good at searching
~Graph engine
Index: spark
Type:tags_time Type:tags_graph

26. Shard
Shard Shard
Shard
Query #1
Elasticsearch cluster

27. spark/tags:
{
”tags”:”kdc:latitude-longitude",
“aws_in_time”:2985,
“aws_no_in_time”:3023,
“num_aws”:29795,
“tol_time":3234324,
“num_no_aws”:796
}
~Efficiency
~Data update
* Response time: the time from each question being posted to getting its first answer
* average response time=tot_time/num_aws
* Prob. of being answered in 10 mins=aws_in_time/(num_aws+num_no_aws)