This document discusses using machine learning and data analysis techniques to improve community management and content moderation on an online question and answer site. It describes building models to automatically rate answers, challenges faced such as lack of training data and complex user behavior, and efforts taken to address these such as collecting more data, using more complex neural network models, and developing tools for moderators. It also covers expanding these techniques to related tasks like question rating, building a taxonomy of tags, and using insights from data and models to further improve the user experience.

1. Due to bad Topics
Back in 2013 ...

2. Party

3. 4.6 Mio.
views per day
60 k
new content items per day
13 Mio.
questions
46 Mio.
answers
Community.

4. Source: AGOF e.V. / internet facts 2013-12
Reach.

5. Hangover

6. Tracking

7. Community Management

8. Tagging

9. Overcome the Reign of Chaos
Michael Stockerl

10. User Behaviour Tracking

11. Track each action
● Pageview
● View of content items
● Click event

12. Remember the user
● ClientId
● Location
● User-Agent

13. The Journey of a Data Point

14. Collect incoming events in Kafka

15. Enrich data points
● Parse arguments
● Join to database ids (e.g. question id)

16. Batch Processing Pipeline

17. Stream Processing Pipeline

18. Learnings
● Kafka is crazy robust
● Parquet accelerates reads
● Spark faster than Hadoop (development and production)
● Backpressure needed for MySQL and Elasticsearch
● Reliable Analytics and A/B Testing pipeline

19. Analytics ONLY with Devs

20. Democratize
Data Access

21. Data Only Available in Primary Stores

22. Make Data Available with Tableau
THRIFT SERVER

23. Learnings
● Tableau easy to learn...
● … but hard to master
● Spark Thriftserver crashes all the time ( somebody same experience? )
● Non - Dev Departments love it

24. Community Management
meets
Machine Learning

25. Manually search for bad
answers

26. Goal: Automatically rate answers
● Sort answers for each question
● Hide bad answers
● Report really bad answers
-> Algorithm to rate answers

27. Solution: Logistic Regression
● Supervised Machine Learning algorithm
● 2000 training & test examples
● Trained first and simple model
● Brought it to production

28. Users were NOT happy

29. Problems
● Model not complex enough
● Similar inputs, different outputs
● Not enough training data
● Missing definition of a good answer

30. Problems
● Model not complex enough
● Similar inputs, different outputs
● Not enough training data
● Missing definition of a good answer
-> Collect more data
-> Use more features
-> More complex model

31. Project Moria

32. Moderation Tool
● Definition of Categories
● Rate answers as fast as possible
● Randomly picked (at first)
● Delete answers

33. Dataset clean?
● See distribution of classification
● Monitor single classifications
● Rate too nice?
● Or too harsh?

34. Project Angmar

35. Project Angmar
● Tried a lot of Supervised Learning Methods
● Feature Engineering: Most crucial part
● Analyse the domain, chart everything

36. Features
Content
length
syntactic complexity
number of links
probability of deletion
Social
votes
most helpful answer
number of comments
answered by expert
Author
gained votes
credibility score
role
ratio of deleted answers
number of answers
number of comments
ratio of reported answers

37. The winner: A simple Neural Net
Answer
vector
AV
normalized
Input
layer
21
3
1
0.2
0.4
0.1
2 0.8
wordCount
voteUp
voteDown
n
Hidden
layer
Output
layer
2n
Score 0.2

38. Result on Testdata

39. Realworld Test
deleted
non-deleted

40. Problems in Production

41. Neural Nets in short

42. Debugging with Radagast
AV
normalized
Input
layer
Hidden
layer
Output
layer
0.2
0.4 -> 0.6
0.1
0.8
wordCount
voteUp
voteDown
n
2n
Score 0.5
+10

43. Before switching models
Amount of answers
for a score range

44. Compare Live and Shadow Model
Amount of answers
for a score range

45. Insights

46. woman
man
Woman write better answers

47. Answerscore per Userlevel
Standard deviation

48. Answerscore per Userlevel
Standard deviation
History of user matters

49. Quality per Tag

50. Outlook

51. Question Rating

52. New Taxonomy

53. 1.005.994 Tags

54. Goal: Directed Acyclic Graph
● Model Hierachy
● Use our content
○ Co-occurence with top tags
○ Repeat with those tags
○ Refinement manually
● Use Cases:
○ Recommender
○ Answer Score
○ Search
○ Experts
Computer
root
Sport
Games Fußball
Fifa 17

55. Prototyp - Related Tags: Umwelt

56. Prototyp - Related Tags: Polizei

57. Prototyp - Related Tags: Herz

58. Prototyp - Related Tags: Wissenschaft

59. Taxonomy
UI any ideas? Computer
root
Sport
Games Fußball
Fifa 17

60. Learning
In God we trust;
all others bring data
W. Edwards Deming

61. Questions?

62. Questions?
We’re hiring!
Scala Finagle
Typescript
Akka-HTTP
Kafka
Spark
MySQL
Elasticsearch
Mesos
Redis