LinkedIn - A Professional Network built with Java Technologies and Agile Practices

LinkedIn: A Professional Social Network Built with Java™ Technologies and Agile Practices Nick Dellamaggiore, Principal Software Engineer Eishay Smith, Senior Software Engineer

Learn how Java™ and agile practices are employed in building large-scale consumer internet sites like LinkedIn, the world’s largest professional network. 2008 JavaOneSM Conference | java.com.sun/javaone | 2

Agenda LinkedIn(troduction) Agile engineering process Architecture Building LinkedIn with Java™ Questions/Discussion 2008 JavaOneSM Conference | java.com.sun/javaone | 3

2008 JavaOneSM Conference | java.com.sun/javaone | 4

LinkedIn Your professional identity on the web The site The numbers • Launched in 2003 • 22 million members • 3 degree network • 4+ million unique visitors/month • Strong focus on privacy • 40 million page views/day The company • 2 million searches/day • Profitable since 2006 • 250K invitations sent/day • 250 employees • 1 million answers posted • Based in Mountain View, CA • We’re hiring! 2008 JavaOneSM Conference | java.com.sun/javaone | 5

Engineering Process Release often • Short, 2-4 week development cycles All dev tasks are broken down into small, manageable engineering cards Strong focus on testing Minimize meetings, status (standup meetings as needed) 2008 JavaOneSM Conference | java.com.sun/javaone | 6

Engineering Cards 2008 JavaOneSM Conference | java.com.sun/javaone | 7

Testing 6500+ unit and integration tests 500 HtmlUnit tests Large-scale Hudson deployment • 20+ nodes • Continuous integration build/smoke test on SVN commit • Nightly test run (full test suite) 2008 JavaOneSM Conference | java.com.sun/javaone | 8

“Celebrities” Test Network 2008 JavaOneSM Conference | java.com.sun/javaone | 9

Hudson 2008 JavaOneSM Conference | java.com.sun/javaone | 10

Testing… Lessons Learned Test suite too focused on integration • Long running time • Long-running setup/teardown routine • Exacerbated by move to SOA • Tests become redundant due to overlapping integration tests • Better: use mocks (EasyMock works great) 2008 JavaOneSM Conference | java.com.sun/javaone | 11

Architecture 2008 JavaOneSM Conference | java.com.sun/javaone | 12

LinkedIn Architecture: 2003-2005 2008 JavaOneSM Conference | java.com.sun/javaone | 13

LinkedIn Architecture: 2006 2008 JavaOneSM Conference | java.com.sun/javaone | 14

LinkedIn Architecture: Today 2008 JavaOneSM Conference | java.com.sun/javaone | 15

LinkedIn Architecture: Today 2008 JavaOneSM Conference | java.com.sun/javaone | 16

Service-oriented Architecture Benefits Caveats • Each service can scale • Diminished data integrity, independently consistency • Encourages decoupling, reduces • No cross-service transactions, code+deployment dependencies referential integrity or joins • Graceful degradation of • Backward compatibility functionality • “8 Fallacies of Distributed Computing” Takeaways • Not applicable to small or simple sites, startups • Build services with interfaces, POJOs, dependency injection • Be conscious of API design, granularity 2008 JavaOneSM Conference | java.com.sun/javaone | 17

Services 2008 JavaOneSM Conference | java.com.sun/javaone | 18

The Cloud Backend server caching the entire LinkedIn Network Graph cache • Updated via Databus • Persisted to disk on shutdown Transient network cache • Computationally intensive to build • Bound to member session • Sticky load balancing to Cloud Graph operations: • findRoute(m1, m2) • visit(visitor, deg) • visit(visitor, deg, since) 2008 JavaOneSM Conference | java.com.sun/javaone | 19

The Cloud The numbers 22M nodes, 120 million edges 12 GB JVM heap • In-memory caches implemented in C++, accessed via JNI 40 Cloud server instances deployed in production 2008 JavaOneSM Conference | java.com.sun/javaone | 20

LinkedIn News Part of a new trend in LinkedIn Crawling the web for news • Distributed task management • Parsing technologies: StAX, Rome Indexing and searching • Serving the right news to the right people Duplications Duplications Duplications • Real time problem, fast solution • 500 articles batch = 125K compares 2008 JavaOneSM Conference | java.com.sun/javaone | 21

News Service Architecture 2008 JavaOneSM Conference | java.com.sun/javaone | 22

LinkedIn: 99% Pure Java™ Plumbing Services Tier Web Tier Tools Spring Remoting LinkedIn Spring Louis (in-house fwk) Hudson (CI) ActiveMQ Eh-cache Spring MVC Eclipse+Mylyn Quartz Lucene Grails JIRA/Greenhopper HttpClient Jetty DWR JUnit, HtmlUnit 2008 JavaOneSM Conference | java.com.sun/javaone | 23

Why we love Java™ at LinkedIn Static typing is a lifesaver • Huge codebase, 1M+ lines of code • Refactoring with confidence • Code navigation (via IDE) • 20 active branches, compiler helps with merging • Scaling the engineering organization • Currently 50+ engineers, 8 teams… and growing • Excellent talent pool of Java™ engineers to hire from Community 2008 JavaOneSM Conference | java.com.sun/javaone | 24

LinkedIn: A Professional Social Network Built with Java™ Technologies and Agile Practices Nick Dellamaggiore (http://www.linkedin.com/in/nick) Eishay Smith (http://www.linkedin.com/in/eishay) We’re hiring!

http://www.insight-it.ru	6529
http://www.dbanotes.net	1953
http://glemir.xplore.cn	748
http://chanwook.tistory.com	699
http://www.dosideas.com	612
http://blog.linkedin.com	586
http://www.slideshare.net	293
http://technofriends.in	179
http://brigomp.blogspot.com	157
http://voituk.kiev.ua	147
http://www.20ju.com	138
http://manuravichandran.com	105
http://confluence.billetel.fr	103
http://localhost	71
http://localhost	71
http://blogs.sun.com	66
http://www.eishay.com	53
http://www.kuqin.com	52
http://dbanotes.net	51
http://blog.csdn.net	49
http://xp-rs.blogspot.com	43
http://www.cnblogs.com	40
http://www.planetacodigo.com	38
http://www.forschung-direkt.eu	36
http://ruzadrca.blogspot.com	33
http://blog.falkayn.com	30
http://athanassioskollyris.blogspot.com	28
http://eltechie.blogspot.com	28
http://www.zhuaxia.com	25
http://eishay.blogspot.com	24
http://www.jroller.com	23
http://static.slideshare.net	23
http://www.vivekjuneja.in	21
http://technos.ouvaton.org	21
http://jujuyjug.blogspot.com	20
http://www.pitako.com.br	19
http://blog.19lou.com	18
http://heeha.wordpress.com	15
http://www.e-presentations.us	13
http://www.xianguo.com	13
http://webcache.googleusercontent.com	13
http://re.xianguo.com	12
http://www.bullog.cn	12
http://www.i2e.com.es	11
http://www.infosertec.com.ar	11
http://www.sysbus.com	10
http://translate.googleusercontent.com	10
https://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	10
http://dl.dropbox.com	10
http://www.tica2.ro	9
http://analyse.org.ua	8

LinkedIn - A Professional Network built with Java Technologies and Agile Practices

by LinkedIn

Accessibility

Categories

Upload Details

Usage Rights

152 Embeds 13,503

Statistics

LinkedIn - A Professional Network built with Java Technologies and Agile Practices Presentation Transcript