SlideShare a Scribd company logo
#twitterflight
The Open Source… 
Behind the Tweets 
October 22, 2014 #twitterflight
Open source is everywhere! 
On your phone, in your car… and within Twitter! 
! 
http://www4.mercedes-benz.com/manual-cars/ba/foss/content/en/assets/FOSS_licences.pdf 
iOS: General->About->Legal->Legal Notices 
! 
Vine: General->About->Legal 
!
Chris Aniszczyk 
Head of Open Source 
@cra
Twitter runs on Open Source
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin
Tweet!
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin 
https://dev.twitter.com/rest/reference/post/statuses/update 
Your first stop as a tweet: Twitter Front End (TFE) 
A fancy reverse proxy for HTTP traffic built on the JVM 
Handles authentication, rate limits and more! 
Powered by the open source project Netty: http://netty.io
Netty at Twitter 
Netty is open source Java NIO framework 
Used heavily at Twitter 
Healthy adopter community: 
http://netty.io/wiki/adopters.html 
! 
Cloudhopper sends billions of SMS messages 
per month using Netty 
https://github.com/twitter/cloudhopper-smpp 
! 
We contributed SPDY support to Netty: 
http://netty.io/news/2012/02/04/3-3-1-spdy.html 
*https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch 
fin 
Twitter backend architecture is *service-oriented (on the JVM) 
Core services are built on top of Finagle (using an API framework) 
Finagle is written in Scala and built on top of Netty 
https://github.com/twitter/finagle 
*http://www.slideshare.net/InfoQ/decomposing-twitter-adventures-in-serviceoriented-architecture
Finagle at Twitter 
Why Scala? 
Scala enables succinct expression (vs Java) 
Less typing is less reading; brevity enhances clarity 
Two open source Scala/Finagle guides from Twitter: 
https://twitter.github.io/effectivescala/ 
https://twitter.github.io/scala_school/ 
! 
Finagle is our fault tolerant protocol-agnostic 
RCP framework built on Netty 
Emphasizes services modularity via async futures 
Handles failover semantics, metrics, logging etc… 
*https://blog.twitter.com/2014/netty-at-twitter-with-finagle
Finagle Service Example 
// #1 Create a client for each service! 
val timelineSvc = Thrift.newIface[TimelineService](...)! 
val tweetSvc = Thrift.newIface[TweetService](...)! 
val authSvc = Thrift.newIface[AuthService](...)! 
! 
// #2 Create new Filter to authenticate incoming requests! 
val authFilter = Filter.mk[Req, AuthReq, Res, Res] { (req, svc) =>! 
authSvc.authenticate(req) flatMap svc(_)! 
}! 
! 
// #3 Create a service to convert an authenticated timeline request to a json response! 
val apiService = Service.mk[AuthReq, Res] { req =>! 
timelineSvc(req.userId) flatMap {tl =>! 
val tweets = tl map tweetSvc.getById(_)! 
Future.collect(tweets) map tweetsToJson(_) }! 
}! 
}! 
! 
// #4 Start a new HTTP server on port 80 using the authenticating filter and our service! 
Http.serve(":80", authFilter andThen apiService)!
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin 
Tweets need to be stored somewhere (via a Finagle-based core service) 
TBird: persistent storage for tweets 
Built originally on Gizzard: https://github.com/twitter/gizzard 
Tweets stored in sharded and replicated MySQL 
TFlock: track relations between users and tweets 
Built originally on FlockDB: https://github.com/twitter/flockdb
MySQL at Twitter 
Maintain a public fork of v5.5/v5.6 
Goal is to“work” with upstream 
https://github.com/twitter/mysql 
Co-founded the WebScaleSQL.org effort
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin 
When a tweet is generated it needs to be written to all relevant timelines 
Timelines are essentially a list of tweet ids (heavily cached) 
Fanout is the process where tweets are delivered to timelines 
For caching we rely on the open source project Redis 
https://github.com/antirez/redis
Redis at Twitter 
Redis is used for caching timelines and more! 
Added custom logging, data structures 
We are working to upstream some changes… 
@thinkingfish gave a fantastic talk on this: 
https://www.youtube.com/watch?v=rP9EKvWt0zo 
! 
Open Source Proxy for Redis: Twemproxy 
https://github.com/twitter/twemproxy 
Used by Vine, Pinterest, Wikimedia, Snapchat etc…
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
Everyone searches for tweets: https://dev.twitter.com/rest/public/search 
In fact, one of the most heavily trafficked search engines in the world 
Back in the day, Twitter search was built on MySQL 
Today, Twitter search is an optimized real-time search/indexing technology 
Powered by Apache Lucene: http://lucene.apache.org 
! 
! 
tweet write fanout search batch fin
Lucene (earlybird) at Twitter 
Earlybird* is Twitter’s real-time search engine 
built on top of Apache Lucene 
! 
We optimized Lucene (cut corners) to handle 
tweets only since that’s all we do 
e.g., less space: 140 characters only need 8 bits 
! 
Read about Blender, our search front-end 
https://blog.twitter.com/2011/twitter-search-now-3x-faster 
*http://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal_ICDE2012.pdf
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch 
Hadoop is used for many things at Twitter, like counting words :) 
scribe logs, batch processing, recommendations, trends, user modeling and more! 
10,000+ hadoop servers, 100,000+ daily hadoop jobs,10M+ daily hadoop tasks 
Parquet is a columnar storage format for Hadoop 
https://parquet.incubator.apache.org 
Scalding is our Scala DSL for writing Hadoop jobs 
https://github.com/twitter/scalding 
! 
! 
fin
Parquet/Scalding at Twitter 
Parquet* is a columnar storage format 
Initially a collaboration between Twitter/Cloudera 
Inspired by Google Dremel paper** 
Now at Apache: http://parquet.incubator.apache.org/ 
! 
Scalding built on top of Scala and Cascading 
https://github.com/Cascading/cascading 
Makes it easier* to write Hadoop jobs (using Scala) 
*https://blog.twitter.com/2013/announcing-parquet-10-columnar-storage-for-hadoop
Scalding Example 
import com.twitter.scalding._! 
! 
// can’t have a Hadoop example without word count!! 
class WordCountJob(args : Args) extends Job(args) {! 
TextLine( args("input") )! 
.flatMap('line -> 'word) { line : String => line.split("""s+""") }! 
.groupBy('word) { _.size }! 
.write( Tsv( args("output") ) )! 
} 
https://github.com/twitter/scalding/wiki/Rosetta-Code
Life of a Tweet 
What open source technology do we use behind the scenes when we tweet? 
tweet write fanout search batch fin
Sharing is caring, contribute! 
Lets all make Twitter better! 
! 
! 
! 
opensource.twitter.com https://github.com/twitter
New Open Source API Samples 
Hack on the samples and improve them! 
https://github.com/twitterdev (t.co/code) 
! 
Also, later today check out 
the lightning talk by Andrew 
Noonan later about the 
“Twitter’s developer toolbox” 
!
Thank You
Q&A 
The Open Source Behind the Tweets 
http://opensource.twitter.com 
! 
Hope you learned something new! 
Come see us at the @TwitterOSS Booth! 
Chris Aniszczyk (@cra)
Resources 
https://opensource.twitter.com 
https://github.com/twitter/finagle 
https://github.com/twitter/zipkin 
https://github.com/twitter/scalding 
https://github.com/twitter/mysql 
https://github.com/twitter/twemproxy 
https://twitter.github.io/scala_school 
http://webscalesql.org 
http://mesos.apache.org 
http://parquet.incubator.apache.org 
!
Backup Slides 
October 22, 2014 #twitterflight
Where does it all run? 
Main concept: Datacenter as a computer 
Aggregation and not virtualization 
! 
! 
! 
mesos.apache.org 
framework 
offer 
hostname 
4 CPUs 
4 GB RAM 
offer 
hostname 
4 CPUs 
4 GB RAM 
offer 
hostname 
4 CPUs 
4 GB RAM 
offer 
hostname 
4 CPUs 
4 GB RAM 
masters aurora.incubator.apache.org
Profiles 
Search / S&R 
Trends / S&R 
Home timeline / TLS 
PTw / Ads 
Compose 
Contact import / 
Growth 
DMs / Social Discover / S&R 
WtF / S&R

More Related Content

What's hot

Contributing to Open Source #Hacktoberfest
Contributing to Open Source #HacktoberfestContributing to Open Source #Hacktoberfest
Contributing to Open Source #Hacktoberfest
Cheah Eng Soon
 
Ruby and Twitter at the Ruby drink-up of Sophia, January 2013
Ruby and Twitter at the Ruby drink-up of Sophia, January 2013Ruby and Twitter at the Ruby drink-up of Sophia, January 2013
Ruby and Twitter at the Ruby drink-up of Sophia, January 2013
rivierarb
 
Open Source: Beyond the Code
Open Source: Beyond the CodeOpen Source: Beyond the Code
Open Source: Beyond the Code
Kerri Shotts
 
Open Source and the Girl Geek (English)
Open Source and the Girl Geek (English)Open Source and the Girl Geek (English)
Open Source and the Girl Geek (English)
Girl Geek Dinners Milano
 
Speedupiosdevelopment
SpeedupiosdevelopmentSpeedupiosdevelopment
Speedupiosdevelopment
slidepaddemo
 
3112 final
3112 final3112 final
3112 final
Matthew Kaiser
 
LA Python #1: Intro, Events, Advocacy
LA Python #1: Intro, Events, AdvocacyLA Python #1: Intro, Events, Advocacy
LA Python #1: Intro, Events, Advocacy
Audrey Roy
 
Hacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginnersHacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginners
DeepikaRana30
 
Hactoberfest presentation
Hactoberfest presentationHactoberfest presentation
Hactoberfest presentation
AITIKDANDAPAT
 
BarCamb Connotea by Ian Mulvany
BarCamb Connotea by Ian MulvanyBarCamb Connotea by Ian Mulvany
BarCamb Connotea by Ian Mulvany
Ian Mulvany
 
What every successful open source project needs
What every successful open source project needsWhat every successful open source project needs
What every successful open source project needs
Steven Francia
 

What's hot (11)

Contributing to Open Source #Hacktoberfest
Contributing to Open Source #HacktoberfestContributing to Open Source #Hacktoberfest
Contributing to Open Source #Hacktoberfest
 
Ruby and Twitter at the Ruby drink-up of Sophia, January 2013
Ruby and Twitter at the Ruby drink-up of Sophia, January 2013Ruby and Twitter at the Ruby drink-up of Sophia, January 2013
Ruby and Twitter at the Ruby drink-up of Sophia, January 2013
 
Open Source: Beyond the Code
Open Source: Beyond the CodeOpen Source: Beyond the Code
Open Source: Beyond the Code
 
Open Source and the Girl Geek (English)
Open Source and the Girl Geek (English)Open Source and the Girl Geek (English)
Open Source and the Girl Geek (English)
 
Speedupiosdevelopment
SpeedupiosdevelopmentSpeedupiosdevelopment
Speedupiosdevelopment
 
3112 final
3112 final3112 final
3112 final
 
LA Python #1: Intro, Events, Advocacy
LA Python #1: Intro, Events, AdvocacyLA Python #1: Intro, Events, Advocacy
LA Python #1: Intro, Events, Advocacy
 
Hacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginnersHacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginners
 
Hactoberfest presentation
Hactoberfest presentationHactoberfest presentation
Hactoberfest presentation
 
BarCamb Connotea by Ian Mulvany
BarCamb Connotea by Ian MulvanyBarCamb Connotea by Ian Mulvany
BarCamb Connotea by Ian Mulvany
 
What every successful open source project needs
What every successful open source project needsWhat every successful open source project needs
What every successful open source project needs
 

Similar to The Open Source... Behind the Tweets

OpenWhisk by Example - Auto Retweeting Example in Python
OpenWhisk by Example - Auto Retweeting Example in PythonOpenWhisk by Example - Auto Retweeting Example in Python
OpenWhisk by Example - Auto Retweeting Example in Python
CodeOps Technologies LLP
 
We are losing our tweets!
We are losing our tweets!We are losing our tweets!
We are losing our tweets!
John O'Brien III
 
Intro to Perfect - LA presentation
Intro to Perfect - LA presentationIntro to Perfect - LA presentation
Intro to Perfect - LA presentation
Tim Taplin
 
API Documentation Workshop tcworld India 2015
API Documentation Workshop tcworld India 2015API Documentation Workshop tcworld India 2015
API Documentation Workshop tcworld India 2015
Tom Johnson
 
iOS勉強会
iOS勉強会iOS勉強会
iOS勉強会
Hiroaki Murayama
 
2022 APIsecure_Securing APIs with Open Standards
2022 APIsecure_Securing APIs with Open Standards2022 APIsecure_Securing APIs with Open Standards
2022 APIsecure_Securing APIs with Open Standards
APIsecure_ Official
 
python programming.pptx
python programming.pptxpython programming.pptx
python programming.pptx
Kaviya452563
 
Social Media Data
Social Media DataSocial Media Data
Social Media Data
Will Simm
 
Building TweetEngine
Building TweetEngineBuilding TweetEngine
Building TweetEngine
ikailan
 
Mining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started GuideMining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started Guide
Matthew Russell
 
BotCommons: Metadata for Bots - Devoxx 2017
BotCommons: Metadata for Bots - Devoxx 2017BotCommons: Metadata for Bots - Devoxx 2017
BotCommons: Metadata for Bots - Devoxx 2017
Cisco DevNet
 
Web 2
Web 2Web 2
Mining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started GuideMining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started Guide
Matthew Russell
 
DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016
DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016
DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016
Cisco DevNet
 
Getting Started with Titanium
Getting Started with TitaniumGetting Started with Titanium
Getting Started with Titanium
Kevin Whinnery
 
Getting Started with Titanium
Getting Started with TitaniumGetting Started with Titanium
Getting Started with Titanium
Axway Appcelerator
 
Flutter - Fluent-Utter
Flutter - Fluent-UtterFlutter - Fluent-Utter
Flutter - Fluent-Utter
Apurva Gupta
 
Embedding Messages and Video Calls in your apps
Embedding Messages and Video Calls in your appsEmbedding Messages and Video Calls in your apps
Embedding Messages and Video Calls in your apps
Cisco DevNet
 
Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...
Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...
Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...
Codemotion
 
Try! Swift Tokyo2017
Try! Swift Tokyo2017Try! Swift Tokyo2017
Try! Swift Tokyo2017
Amy Cheong
 

Similar to The Open Source... Behind the Tweets (20)

OpenWhisk by Example - Auto Retweeting Example in Python
OpenWhisk by Example - Auto Retweeting Example in PythonOpenWhisk by Example - Auto Retweeting Example in Python
OpenWhisk by Example - Auto Retweeting Example in Python
 
We are losing our tweets!
We are losing our tweets!We are losing our tweets!
We are losing our tweets!
 
Intro to Perfect - LA presentation
Intro to Perfect - LA presentationIntro to Perfect - LA presentation
Intro to Perfect - LA presentation
 
API Documentation Workshop tcworld India 2015
API Documentation Workshop tcworld India 2015API Documentation Workshop tcworld India 2015
API Documentation Workshop tcworld India 2015
 
iOS勉強会
iOS勉強会iOS勉強会
iOS勉強会
 
2022 APIsecure_Securing APIs with Open Standards
2022 APIsecure_Securing APIs with Open Standards2022 APIsecure_Securing APIs with Open Standards
2022 APIsecure_Securing APIs with Open Standards
 
python programming.pptx
python programming.pptxpython programming.pptx
python programming.pptx
 
Social Media Data
Social Media DataSocial Media Data
Social Media Data
 
Building TweetEngine
Building TweetEngineBuilding TweetEngine
Building TweetEngine
 
Mining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started GuideMining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started Guide
 
BotCommons: Metadata for Bots - Devoxx 2017
BotCommons: Metadata for Bots - Devoxx 2017BotCommons: Metadata for Bots - Devoxx 2017
BotCommons: Metadata for Bots - Devoxx 2017
 
Web 2
Web 2Web 2
Web 2
 
Mining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started GuideMining the Social Web for Fun and Profit: A Getting Started Guide
Mining the Social Web for Fun and Profit: A Getting Started Guide
 
DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016
DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016
DevNet @TAG - Spark & Tropo APIs - Milan/Rome May 2016
 
Getting Started with Titanium
Getting Started with TitaniumGetting Started with Titanium
Getting Started with Titanium
 
Getting Started with Titanium
Getting Started with TitaniumGetting Started with Titanium
Getting Started with Titanium
 
Flutter - Fluent-Utter
Flutter - Fluent-UtterFlutter - Fluent-Utter
Flutter - Fluent-Utter
 
Embedding Messages and Video Calls in your apps
Embedding Messages and Video Calls in your appsEmbedding Messages and Video Calls in your apps
Embedding Messages and Video Calls in your apps
 
Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...
Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...
Steve Sfartz - How to embed Messaging and Video in your apps - Codemotion Mil...
 
Try! Swift Tokyo2017
Try! Swift Tokyo2017Try! Swift Tokyo2017
Try! Swift Tokyo2017
 

More from Chris Aniszczyk

Bringing an open source project to the Linux Foundation
Bringing an open source project to the Linux FoundationBringing an open source project to the Linux Foundation
Bringing an open source project to the Linux Foundation
Chris Aniszczyk
 
Starting an Open Source Program Office (OSPO)
Starting an Open Source Program Office (OSPO)Starting an Open Source Program Office (OSPO)
Starting an Open Source Program Office (OSPO)
Chris Aniszczyk
 
Open Container Initiative Update
Open Container Initiative UpdateOpen Container Initiative Update
Open Container Initiative Update
Chris Aniszczyk
 
Cloud Native Landscape (CNCF and OCI)
Cloud Native Landscape (CNCF and OCI)Cloud Native Landscape (CNCF and OCI)
Cloud Native Landscape (CNCF and OCI)
Chris Aniszczyk
 
Rise of Open Source Programs
Rise of Open Source ProgramsRise of Open Source Programs
Rise of Open Source Programs
Chris Aniszczyk
 
The Open Container Initiative (OCI) at 12 months
The Open Container Initiative (OCI) at 12 monthsThe Open Container Initiative (OCI) at 12 months
The Open Container Initiative (OCI) at 12 months
Chris Aniszczyk
 
Open Source Lessons from the TODO Group
Open Source Lessons from the TODO GroupOpen Source Lessons from the TODO Group
Open Source Lessons from the TODO Group
Chris Aniszczyk
 
Getting Students Involved in Open Source
Getting Students Involved in Open SourceGetting Students Involved in Open Source
Getting Students Involved in Open Source
Chris Aniszczyk
 
Apache Mesos at Twitter (Texas LinuxFest 2014)
Apache Mesos at Twitter (Texas LinuxFest 2014)Apache Mesos at Twitter (Texas LinuxFest 2014)
Apache Mesos at Twitter (Texas LinuxFest 2014)
Chris Aniszczyk
 
Evolution of The Twitter Stack
Evolution of The Twitter StackEvolution of The Twitter Stack
Evolution of The Twitter Stack
Chris Aniszczyk
 
Open Source Compliance at Twitter
Open Source Compliance at TwitterOpen Source Compliance at Twitter
Open Source Compliance at Twitter
Chris Aniszczyk
 
Effective Development With Eclipse Mylyn, Git, Gerrit and Hudson
Effective Development With Eclipse Mylyn, Git, Gerrit and HudsonEffective Development With Eclipse Mylyn, Git, Gerrit and Hudson
Effective Development With Eclipse Mylyn, Git, Gerrit and Hudson
Chris Aniszczyk
 
Effective Git with Eclipse
Effective Git with EclipseEffective Git with Eclipse
Effective Git with Eclipse
Chris Aniszczyk
 
Evolution of Version Control In Open Source
Evolution of Version Control In Open SourceEvolution of Version Control In Open Source
Evolution of Version Control In Open Source
Chris Aniszczyk
 
ESE 2010: Using Git in Eclipse
ESE 2010: Using Git in EclipseESE 2010: Using Git in Eclipse
ESE 2010: Using Git in Eclipse
Chris Aniszczyk
 
SWTBot Tutorial
SWTBot TutorialSWTBot Tutorial
SWTBot Tutorial
Chris Aniszczyk
 
Helios in Action: Git at Eclipse
Helios in Action: Git at EclipseHelios in Action: Git at Eclipse
Helios in Action: Git at Eclipse
Chris Aniszczyk
 
Introduction to EclipseRT (JAX 2010)
Introduction to EclipseRT (JAX 2010)Introduction to EclipseRT (JAX 2010)
Introduction to EclipseRT (JAX 2010)
Chris Aniszczyk
 
EclipseRT, Equinox and OSGi
EclipseRT, Equinox and OSGiEclipseRT, Equinox and OSGi
EclipseRT, Equinox and OSGi
Chris Aniszczyk
 
Open Source From The Trenches: How to Get Involved with Open Source and be Su...
Open Source From The Trenches: How to Get Involved with Open Source and be Su...Open Source From The Trenches: How to Get Involved with Open Source and be Su...
Open Source From The Trenches: How to Get Involved with Open Source and be Su...
Chris Aniszczyk
 

More from Chris Aniszczyk (20)

Bringing an open source project to the Linux Foundation
Bringing an open source project to the Linux FoundationBringing an open source project to the Linux Foundation
Bringing an open source project to the Linux Foundation
 
Starting an Open Source Program Office (OSPO)
Starting an Open Source Program Office (OSPO)Starting an Open Source Program Office (OSPO)
Starting an Open Source Program Office (OSPO)
 
Open Container Initiative Update
Open Container Initiative UpdateOpen Container Initiative Update
Open Container Initiative Update
 
Cloud Native Landscape (CNCF and OCI)
Cloud Native Landscape (CNCF and OCI)Cloud Native Landscape (CNCF and OCI)
Cloud Native Landscape (CNCF and OCI)
 
Rise of Open Source Programs
Rise of Open Source ProgramsRise of Open Source Programs
Rise of Open Source Programs
 
The Open Container Initiative (OCI) at 12 months
The Open Container Initiative (OCI) at 12 monthsThe Open Container Initiative (OCI) at 12 months
The Open Container Initiative (OCI) at 12 months
 
Open Source Lessons from the TODO Group
Open Source Lessons from the TODO GroupOpen Source Lessons from the TODO Group
Open Source Lessons from the TODO Group
 
Getting Students Involved in Open Source
Getting Students Involved in Open SourceGetting Students Involved in Open Source
Getting Students Involved in Open Source
 
Apache Mesos at Twitter (Texas LinuxFest 2014)
Apache Mesos at Twitter (Texas LinuxFest 2014)Apache Mesos at Twitter (Texas LinuxFest 2014)
Apache Mesos at Twitter (Texas LinuxFest 2014)
 
Evolution of The Twitter Stack
Evolution of The Twitter StackEvolution of The Twitter Stack
Evolution of The Twitter Stack
 
Open Source Compliance at Twitter
Open Source Compliance at TwitterOpen Source Compliance at Twitter
Open Source Compliance at Twitter
 
Effective Development With Eclipse Mylyn, Git, Gerrit and Hudson
Effective Development With Eclipse Mylyn, Git, Gerrit and HudsonEffective Development With Eclipse Mylyn, Git, Gerrit and Hudson
Effective Development With Eclipse Mylyn, Git, Gerrit and Hudson
 
Effective Git with Eclipse
Effective Git with EclipseEffective Git with Eclipse
Effective Git with Eclipse
 
Evolution of Version Control In Open Source
Evolution of Version Control In Open SourceEvolution of Version Control In Open Source
Evolution of Version Control In Open Source
 
ESE 2010: Using Git in Eclipse
ESE 2010: Using Git in EclipseESE 2010: Using Git in Eclipse
ESE 2010: Using Git in Eclipse
 
SWTBot Tutorial
SWTBot TutorialSWTBot Tutorial
SWTBot Tutorial
 
Helios in Action: Git at Eclipse
Helios in Action: Git at EclipseHelios in Action: Git at Eclipse
Helios in Action: Git at Eclipse
 
Introduction to EclipseRT (JAX 2010)
Introduction to EclipseRT (JAX 2010)Introduction to EclipseRT (JAX 2010)
Introduction to EclipseRT (JAX 2010)
 
EclipseRT, Equinox and OSGi
EclipseRT, Equinox and OSGiEclipseRT, Equinox and OSGi
EclipseRT, Equinox and OSGi
 
Open Source From The Trenches: How to Get Involved with Open Source and be Su...
Open Source From The Trenches: How to Get Involved with Open Source and be Su...Open Source From The Trenches: How to Get Involved with Open Source and be Su...
Open Source From The Trenches: How to Get Involved with Open Source and be Su...
 

Recently uploaded

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Zilliz
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
TIPNGVN2
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Paige Cruz
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Zilliz
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 

Recently uploaded (20)

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 

The Open Source... Behind the Tweets

  • 2. The Open Source… Behind the Tweets October 22, 2014 #twitterflight
  • 3. Open source is everywhere! On your phone, in your car… and within Twitter! ! http://www4.mercedes-benz.com/manual-cars/ba/foss/content/en/assets/FOSS_licences.pdf iOS: General->About->Legal->Legal Notices ! Vine: General->About->Legal !
  • 4. Chris Aniszczyk Head of Open Source @cra
  • 5. Twitter runs on Open Source
  • 6. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin
  • 8. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin https://dev.twitter.com/rest/reference/post/statuses/update Your first stop as a tweet: Twitter Front End (TFE) A fancy reverse proxy for HTTP traffic built on the JVM Handles authentication, rate limits and more! Powered by the open source project Netty: http://netty.io
  • 9. Netty at Twitter Netty is open source Java NIO framework Used heavily at Twitter Healthy adopter community: http://netty.io/wiki/adopters.html ! Cloudhopper sends billions of SMS messages per month using Netty https://github.com/twitter/cloudhopper-smpp ! We contributed SPDY support to Netty: http://netty.io/news/2012/02/04/3-3-1-spdy.html *https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead
  • 10. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin Twitter backend architecture is *service-oriented (on the JVM) Core services are built on top of Finagle (using an API framework) Finagle is written in Scala and built on top of Netty https://github.com/twitter/finagle *http://www.slideshare.net/InfoQ/decomposing-twitter-adventures-in-serviceoriented-architecture
  • 11. Finagle at Twitter Why Scala? Scala enables succinct expression (vs Java) Less typing is less reading; brevity enhances clarity Two open source Scala/Finagle guides from Twitter: https://twitter.github.io/effectivescala/ https://twitter.github.io/scala_school/ ! Finagle is our fault tolerant protocol-agnostic RCP framework built on Netty Emphasizes services modularity via async futures Handles failover semantics, metrics, logging etc… *https://blog.twitter.com/2014/netty-at-twitter-with-finagle
  • 12. Finagle Service Example // #1 Create a client for each service! val timelineSvc = Thrift.newIface[TimelineService](...)! val tweetSvc = Thrift.newIface[TweetService](...)! val authSvc = Thrift.newIface[AuthService](...)! ! // #2 Create new Filter to authenticate incoming requests! val authFilter = Filter.mk[Req, AuthReq, Res, Res] { (req, svc) =>! authSvc.authenticate(req) flatMap svc(_)! }! ! // #3 Create a service to convert an authenticated timeline request to a json response! val apiService = Service.mk[AuthReq, Res] { req =>! timelineSvc(req.userId) flatMap {tl =>! val tweets = tl map tweetSvc.getById(_)! Future.collect(tweets) map tweetsToJson(_) }! }! }! ! // #4 Start a new HTTP server on port 80 using the authenticating filter and our service! Http.serve(":80", authFilter andThen apiService)!
  • 13. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin
  • 14. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin Tweets need to be stored somewhere (via a Finagle-based core service) TBird: persistent storage for tweets Built originally on Gizzard: https://github.com/twitter/gizzard Tweets stored in sharded and replicated MySQL TFlock: track relations between users and tweets Built originally on FlockDB: https://github.com/twitter/flockdb
  • 15. MySQL at Twitter Maintain a public fork of v5.5/v5.6 Goal is to“work” with upstream https://github.com/twitter/mysql Co-founded the WebScaleSQL.org effort
  • 16. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin
  • 17. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin When a tweet is generated it needs to be written to all relevant timelines Timelines are essentially a list of tweet ids (heavily cached) Fanout is the process where tweets are delivered to timelines For caching we rely on the open source project Redis https://github.com/antirez/redis
  • 18. Redis at Twitter Redis is used for caching timelines and more! Added custom logging, data structures We are working to upstream some changes… @thinkingfish gave a fantastic talk on this: https://www.youtube.com/watch?v=rP9EKvWt0zo ! Open Source Proxy for Redis: Twemproxy https://github.com/twitter/twemproxy Used by Vine, Pinterest, Wikimedia, Snapchat etc…
  • 19. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin
  • 20. Life of a Tweet What open source technology do we use behind the scenes when we tweet? Everyone searches for tweets: https://dev.twitter.com/rest/public/search In fact, one of the most heavily trafficked search engines in the world Back in the day, Twitter search was built on MySQL Today, Twitter search is an optimized real-time search/indexing technology Powered by Apache Lucene: http://lucene.apache.org ! ! tweet write fanout search batch fin
  • 21. Lucene (earlybird) at Twitter Earlybird* is Twitter’s real-time search engine built on top of Apache Lucene ! We optimized Lucene (cut corners) to handle tweets only since that’s all we do e.g., less space: 140 characters only need 8 bits ! Read about Blender, our search front-end https://blog.twitter.com/2011/twitter-search-now-3x-faster *http://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal_ICDE2012.pdf
  • 22. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin
  • 23. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch Hadoop is used for many things at Twitter, like counting words :) scribe logs, batch processing, recommendations, trends, user modeling and more! 10,000+ hadoop servers, 100,000+ daily hadoop jobs,10M+ daily hadoop tasks Parquet is a columnar storage format for Hadoop https://parquet.incubator.apache.org Scalding is our Scala DSL for writing Hadoop jobs https://github.com/twitter/scalding ! ! fin
  • 24. Parquet/Scalding at Twitter Parquet* is a columnar storage format Initially a collaboration between Twitter/Cloudera Inspired by Google Dremel paper** Now at Apache: http://parquet.incubator.apache.org/ ! Scalding built on top of Scala and Cascading https://github.com/Cascading/cascading Makes it easier* to write Hadoop jobs (using Scala) *https://blog.twitter.com/2013/announcing-parquet-10-columnar-storage-for-hadoop
  • 25. Scalding Example import com.twitter.scalding._! ! // can’t have a Hadoop example without word count!! class WordCountJob(args : Args) extends Job(args) {! TextLine( args("input") )! .flatMap('line -> 'word) { line : String => line.split("""s+""") }! .groupBy('word) { _.size }! .write( Tsv( args("output") ) )! } https://github.com/twitter/scalding/wiki/Rosetta-Code
  • 26. Life of a Tweet What open source technology do we use behind the scenes when we tweet? tweet write fanout search batch fin
  • 27. Sharing is caring, contribute! Lets all make Twitter better! ! ! ! opensource.twitter.com https://github.com/twitter
  • 28. New Open Source API Samples Hack on the samples and improve them! https://github.com/twitterdev (t.co/code) ! Also, later today check out the lightning talk by Andrew Noonan later about the “Twitter’s developer toolbox” !
  • 30. Q&A The Open Source Behind the Tweets http://opensource.twitter.com ! Hope you learned something new! Come see us at the @TwitterOSS Booth! Chris Aniszczyk (@cra)
  • 31. Resources https://opensource.twitter.com https://github.com/twitter/finagle https://github.com/twitter/zipkin https://github.com/twitter/scalding https://github.com/twitter/mysql https://github.com/twitter/twemproxy https://twitter.github.io/scala_school http://webscalesql.org http://mesos.apache.org http://parquet.incubator.apache.org !
  • 32. Backup Slides October 22, 2014 #twitterflight
  • 33. Where does it all run? Main concept: Datacenter as a computer Aggregation and not virtualization ! ! ! mesos.apache.org framework offer hostname 4 CPUs 4 GB RAM offer hostname 4 CPUs 4 GB RAM offer hostname 4 CPUs 4 GB RAM offer hostname 4 CPUs 4 GB RAM masters aurora.incubator.apache.org
  • 34. Profiles Search / S&R Trends / S&R Home timeline / TLS PTw / Ads Compose Contact import / Growth DMs / Social Discover / S&R WtF / S&R