LinkedIn uses Hadoop and other big data tools to analyze data from its over 400 million user profiles. It focuses on data science techniques like recommendation engines to build products that suggest connections, jobs, and skills comparisons to users. LinkedIn embeds data scientists with product teams and uses an AI Academy to equip employees to optimize AI experiences for members. It analyzes user data like profiles, activities, likes and searches to build a comprehensive understanding of professionals and their networks.

1. Analytics in Action
How LinkedIn uses Hadoop to leverage Big Data
Analytics?
February 2018
Group 1
Darshan Sajeev – 17009
Sumanth J Kamath - 17039

2. Introduction
©2016 L. SCHLENKER
Structure
What is the organization’s business model?
Why does the organization focus in data?
AI Roadmap
Data Science technique which favours the
organization
Link between data science and decision making

3. • What is the organization’s
business model?
• How does the organization use
data strategically?
• How can you apply the AI
Roadmap to a specific challenge
in this industry?
• Which data science techniques
does the organization favor ?
• What is the link between data
science and decision making?
Case Study Questions
Technology

4. • More than
• 400 million profiles
• Across 200+ countries
• More than 100 million monthly visitors
• 3 million company pages
• 2 new members joining every second
• 5.7 billion professional searches
• - LinkedIn is the largest social network for
professionals.
Introduction
Technology

5. Reference
Business Model
Technology

7. Reference
• LinkedIn uses data for its
recommendation engine to build various
data products.
• The data from user profiles and various
network activities is used to build a
comprehensive picture of a member
and his/her connections.
• LinkedIn want to know whom you
should connect with, where you should
apply for a job and how your skills stack
up against your peers as you look for
your dream job.
Why Data Science ?
Technology

8. Reference
Impact on Decision Making
Technology

9. • User generated data - worldwide
• Cookies
• Analysing the ‘Likes’ through which they can predict the traits
of an indivudual user.
Data Sources
Technology

10. Hadoop
Pig
Hive
Azkaban (Workflow)
Avro Data
Zookeeper
Data In- Apache Kafka
Data Out- Apache Kafka and Voldemort
Data Tools
Technology

11. • To scale AI engineers, statisticians, and data scientists, LinkedIn has
adopted a centralized organizational model that embeds their experts
with product teams but maintains the reporting relationship within a
centralized AI organization.
• This allows them to find unique opportunities to cross-collaborate and
problem solve for the entire member experience, while still applying
more localized optimizations for machine learning problems at the
product level.
• LinkedIn AI Academy is another program that helps equip their
employees across the company—in areas like engineering, product
management, etc.—with the knowledge they need to optimally deliver
impactful AI experiences to the members.
Data Science Team
Technology

12. • Voldemort – A NoSQL distributed key value storage system.
• Giraph – Used for social graph computations and
interpretations on Hadoop clusters.
• White Elephant – Parses Hadoop logs and provides
visualization dashboard for Hadoop clusters.
• Decomposer – Contains large matrix decomposition
algorithms implemented in Java.
• Azkaban – Open source workflow system for Hadoop.
• Kafka – Kafka at LinkedIn is used for tracking hundereds of
different events like page views, profile views, network
updates, impressions, logins and searches over a billion
records every day.
• Avatar – LinkedIn’s scalable and highly available OLAP system
used in “Who’s Viewed My Profile” feature.
Data Science TechniquesIntroduction

13. • LinkedIn uses AI technologies and tools to innovate and make very effective
products like
• 1. People You May Know
• 2. Skill Endorsements
• 3. Jobs You May Be Interested In
• 4. News Feed Updates
• Thus with these kinds of innovative products LinkedIn could become the
world’s most reliable professional network. Thereby it could become the
market leader it’s segment with revenue of $780 Million and earnings of 78
cents per share
Lessons learned
Technology

14. https://www.dezyre.com/article/how-linkedin-uses-
hadoop-to-leverage-big-data-analytics/229
https://fourweekmba.com/multi-sided-segment-
business-model/
https://engineering.linkedin.com/blog/2018/10/an-
introduction-to-ai-at-linkedin
Bibliography
Next Steps