Building LinkedIn's Next Generation  Architecture with OSGi
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Building LinkedIn's Next Generation Architecture with OSGi

on

  • 21,754 views

 

Statistics

Views

Total Views
21,754
Views on SlideShare
15,367
Embed Views
6,387

Actions

Likes
37
Downloads
593
Comments
2

38 Embeds 6,387

http://java.dzone.com 3546
http://www.javaplex.com 1548
http://singztechmusings.wordpress.com 637
http://blog.linkedin.com 195
http://michaelsheen.sinaapp.com 98
http://heeha.wordpress.com 61
http://osgi.dzone.com 46
http://www.scoop.it 44
http://www.slideshare.net 44
http://architects.dzone.com 32
http://linkedin.typepad.com 30
http://bouraouibilel.blogspot.com 15
https://wiki.ecmc.hu 13
http://jisi.dreamblog.jp 13
http://www.techgig.com 12
http://bouraouibilel.blogspot.kr 11
http://bouraouibilel.blogspot.fr 9
http://webcache.googleusercontent.com 4
http://translate.googleusercontent.com 4
http://bouraouibilel.blogspot.de 3
http://bouraouibilel.blogspot.jp 2
http://bouraouibilel.blogspot.co.uk 2
http://www.schoox.com 2
http://www.schoox.com 2
http://64.233.183.104 1
http://bouraouibilel.blogspot.be 1
http://paper.li 1
http://www.365dailyjournal.com 1
http://bouraouibilel.blogspot.com.es 1
http://bouraouibilel.blogspot.sk 1
http://bouraouibilel.blogspot.pt 1
https://confluence.ecmc.hu 1
http://www.dzone.com 1
http://jmmiddleware.wordpress.com 1
https://singztechmusings.wordpress.com 1
http://bouraouibilel.blogspot.com.ar 1
http://static.slideshare.net 1
http://www.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Thanks for the presentation. I was not at all familiar with OSGi architecture, now I am incredibly curious. Will be doing quite a bit of research because of this. Excellent presentation.
    Are you sure you want to
    Your message goes here
    Processing…
  • Nice presentation, Yan!
    Regarding slide 27. Distributed OSGi doesn't explicitly state how load balancing should be done, but the idea is that implementors of DOSGi could provide mechanisms that would achieve this behind the scenes. This could be exposed via an intent that expresses that a service is 'load balanced' or 'highly available' where the actual details of how this is exactly done is left up to the implementation of DOSGi used.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Building LinkedIn's Next Generation Architecture with OSGi Presentation Transcript

  • 1. OSGi Building LinkedIn's Next Generation Architecture with OSGI Yan Pujante Distinguished Software Engineer Member of the Founding Team @ LinkedIn ypujante@linkedin.com http://www.linkedin.com/in/yan
  • 2. Background ■ LinkedIn created in 03/2003 • close to 30M members • profitable with 5 revenue lines • HQ in Mountain View (400 employees) ■ Technologies • 2 datacenters (~ 600 machines) • SOA, java, tomcat, spring framework, http, Oracle, MySQL, servlets, jsp, graph, OSGi... • dev on MacOS X, prod on Solaris
  • 3. Challenges ■ Growing engineering team on a monolithic code base (still modular) ■ Growing product team wanting more and more feature, faster ■ Growing operations team deploying more and more servers
  • 4. Challenges (SOA in dev) Front-End Back-End X 1 3 webapp Z 2 Y webapp 4 M N webapp jetty (1) webapp P N R webapp webapp webapp tomcat jetty (2) ■ Front-end: many BL services in one webapp (in tomcat) ■ Back-end: many wars in 5 containers (jetty) with 1 service per war ■ Note that prod and dev are different
  • 5. Container Challenge ■ 1 war with N services • does not scale for dev (conflicts, monolithic...) ■ N wars with 1 service • does not scale from a container point of view (no shared jars) ■ Can add containers but only 12Gb of RAM :(
  • 6. Challenges (SOA in prod) ■ Upgrading back-end service to new version requires downtime • Hardware LB does not account for version ■ Upgrading front-end service to new version requires redeploy • N services in 1 war ■ Adding new back-end service painful • lots of manual configuration (LB, front- end...)
  • 7. Solution: OSGi ■ Belief that OSGi is one of the tools that will help us address those challenges Why OSGi?
  • 8. OSGi intro ■ OSGi means “Open Services Gateway initiative” ■ OSGi is a spec with several implementations • Equinox (eclipse!) • Knoplerfish • Felix (Apache)
  • 9. OSGi intro (cont.) ■ OSGi has some very good properties • Smart class loading => multiple versions jars ok • Highly dynamic => deploy / undeploy built in • Services registry • Highly extensible / configurable ■ OSGi bundle = Jar File with OSGi manifests
  • 10. OSGi bundle Specification-Title: Colorado Software Summit 2008 Hello Wolrd Client Bundle-Activator: com.linkedin.colorado.helloworld.client.HelloWorldClientActivator Import-Package: com.linkedin.colorado.helloworld.api;version=quot;[1.0.0,1.0.1)quot;, com.linkedin.colorado.helloworld.client;version=quot;[1.0.0,1.0.1)quot;, org.osgi.framework;version=quot;[1.4.0,1.4.1)quot; Export-Package: com.linkedin.colorado.helloworld.client;version=quot;1.0.0quot; Bundle-Version: 1.0.0 Bundle-Name: colorado-helloworld-client Bundle-ManifestVersion: 2 Bundle-SymbolicName: colorado-helloworld-client Bnd-LastModified: 1224557973403 Generated-On: 20081020 Tool: Bnd-<unknown version> Implementation-Version: DevBuild
  • 11. BundleActivator ■ In a war: ServletContextListener public interface ServletContextListener extends EventListener { void contextInitialized(ServletContextEvent event); void contextDestroyed(ServletContextEvent event); } web.xml <listener> <listener-class>com.colorado.MyListener</listener-class> </listener> ■ In a Bundle: BundleActivator public interface BundleActivator { void start(BundleContext bundleContext) throws Exception; void stop(BundleContext bundleContext) throws Exception; } MANIFEST.MF Bundle-Activator: com.colorado.MyActivator
  • 12. BundleContext ■ The bundle context is the entry point: • registerService / getServiceReference => to register / get services from registry • addBundleListener / addServiceListener => to listens / extends the framework • even installBundle ! • ... and more
  • 13. Demo ■ Equinox demo... stay tuned...
  • 14. Web Container JARs WAR ■ Services exported Services with Spring/RPC ■ Services in same war can see each other SpringRPC ■ Share the jars WEB Container VM HTTP
  • 15. Web Container JARs JARs Services Services ■ Cannot share jars SpringRPC SpringRPC ■ Cannot talk directly WEB Container VM HTTP
  • 16. OSGi container Bundles ■ Bundles (jars) shared Service Registry ■ Services shared ■ Dynamic replacement of bundles OSGi Container VM • OSGi solves the container challenge
  • 17. Distributed OSGi ■ OSGi is currently container centric ■ Next version (4.2) will introduce distributed OSGi • What do we do in the meantime ?
  • 18. LinkedIn distributed OSGi Bundles Service Service Registry Registry C1 VM ■ Service ‘red’ talks to service ‘green’ (local) • What happens if we move ‘green’ to a different container ?
  • 19. LinkedIn distributed OSGi Bundles Service Registry Bundles C2 Service Registry VM Bundles C1 VM Service Registry C3 VM
  • 20. LinkedIn distributed OSGi Bundles Service Registry Bundles 2 C2 Service Registry VM 2 Bundles C1 VM Service Registry C3 VM
  • 21. LinkedIn distributed OSGi Bundles Service Registry Bundles 2 C2 Service Registry VM 2 3 Bundles C1 VM Service Registry 3 C3 VM
  • 22. LinkedIn distributed OSGi Bundles Service Registry Bundles 2 C2 Service Registry VM 2 3 Bundles C1 VM Service Registry 3 C3 VM
  • 23. LinkedIn distributed OSGi ■ no more N-1 / 1-N problem • libraries and services can be shared in 1 container (mem. footprint smaller) • services shared across containers ■ transparent location to clients • no more configurations to change when adding/removing/moving services ■ software load balancer • much more flexible (handle version)
  • 24. LB and version Bundles Bundles Service Registry Service Registry 2 2 C2 VM 3 Bundles C1 VM Service Registry 3 C3 VM
  • 25. Everything is perfect... ■ ...Not really • OSGi has some problems
  • 26. OSGi problems ■ OSGi is great, but tooling is not quite there yet ■ Not every lib is a bundle ■ OSGi was designed for embedded devices => using it for server side is recent (but very active)
  • 27. OSGi problems ■ OSGi is quite low level => but there is some work to ‘hide’ the complexity • spring DM (spring dynamic module) • multiple vendors containers ■ Spring DM Server, Glassfish, Infiniflow ■ OSGi is container centric, but next version will add distributed OSGi (although no support for LB)
  • 28. OSGi problems (cont.) ■ version management can be nightmarish • Import-Package: com.linkedin.util;version=1.0.0 ■ version=1.0.0 means [1.0.0, ∞) ■ should use at least: version=[1.0.0,2.0.0) ■ runtime is dynamic and can result in untested code being executed!
  • 29. Runtime version problem ■ client1 and client2 uses Service 3.0.0 ■ Dependencies are ‘loose’: [3.0.0, 4.0.0)
  • 30. Runtime version problem ■ We upgrade client1 and service ■ client2 starts using green bundles ■ => Most likely untested
  • 31. Runtime version solutions ? ■ A solution is to lock version: • [1.0.0,1.0.0] => no slack • [1.0.0,1.0.1) => tiny bit of slack for emergency mode ■ Does it really work ? spring 2.5.5 imports hibernate version [3.1,∞) ■ Can OSGi even resolve with everything locked down ?
  • 32. Runtime version problem ■ With maximal lock-down, client2 cannot use the new service
  • 33. Runtime version solutions ? ■ Services cannot talk to each other directly => need serialization for backward compatible services (dynamic proxy)
  • 34. Despite problems... ■ ...OSGi and community moving in the right direction
  • 35. Where are we ? ■ Bundle repository (big amount of work... in progress) ■ Use ivy/bnd for generating bundles ■ Evaluating container • Infiniflow (from Paremus) most likely the one • LinkedIn spring / SCA to deploy composites ■ Work on the load balancer/distribution (in progress) ■ Work on tooling / build integration (Sigil from Paremus) ■ Much much more to do...
  • 36. Conclusion ■ Using OSGi... definitely! ■ But we do not want application developers to have to deal with it.
  • 37. More information ■ Check my posts on the LinkedIn blog (OSGi series). More to come...