Social Shovel is a project to analyzing social network streams in a real time manner.

1. Social Shovel
We shovel Social Networks

2. Team
Soroosh Sarabadani
Mikhail Litvin
Dmytro Zharkov
Vasanth Venkatraman
Frank Honza
Markus Görlich

3. Features
● Monitoring social activity.
● Collect and analyze data.
● Show client’s satisfaction and opinion.
● Find and distinguish social interests and
trends.
● History and statistics.

4. Non functional requirements
● Scalable
● Realtime
● Cloud Friendly

5. Solution
● Message Driven Architecture
● Push Notifications
● Stateless Lightweight Microservices

6. Live Demo

7. Architecture from 2k feet
Social Networks
Stream Readers
Observable
Memory
Classifier
Storage
Repository
Broadcaster

8. So Far So GOOOOOD
● Observable Memory
o Apache ActiveMQ (STOMP)
● Storage
o MongoDB
● Classifier
o scikit-learn
o Natural Language Toolkit
o STOMP

9. Message Persistence
● NoSQL - MongoDB
● Simple message queue
driven storage.
● Provides aggregated
results to frontend.
o Multi criteria support
msg queue
Listener
MongoDB
readers
Frontend

10. Classifier
Learning
Training Lists
(manually “digged” and categorized)
- Twitter (Sentiment)
- Wikipedia (Category)
Observable
Memory
Classification
Startup

11. Classification classes
Satisfaction:
"UNSATISFIED"
"SATISFIED"
"NEUTRAL"
Categories:
"UNDERWEAR"
"ACCESSORIES"
"DRESS"
"JACKET"
"SHIRT"
"SHOE"
"SKIRT"
"SUIT"
"TROUSER"
"NEUTRAL"
Customer reaction via Twitter
Wikipedia: types
“Distinct” Brands
Explicit model names (e.g. Nike Air)

12. Classifier / Data Preparation
- SVM with linear kernel
- multiclass SVM with 1 vs 1 approach
- minimum probability to create “neutral” class

13. Classifier / Tokenizing
- stemming (Porter)
- construction of unigrams + bigrams
- feature for smileys (good and bad ;-)
- tf-idf token weights

14. Front End
HTML5/CSS3
Bootstrap 3
JavaScript
AngularJS
Websockets
ChartsJS
Yeoman/Grunt/Bower

15. Thank you