Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Journey from Monolith to a Modularized Application - Approach and Key Learnings - E Xavier & T Ward


Published on

OSGi Community Event 2017 Presentation by Emanoel Xavier (Intel) and Tim Ward (Paremus)

Open Security Controller (OSC) aims to increase the security of workloads running in cloud and data centers by orchestrating security virtual network functions (VNFs). To achieve that, one of the main design goals of OSC is to allow vendor customization while keeping its core code neutral. Additionally, as an open source project, another import design principal for OSC is code habitability. Evolving from a legacy monolithic application to a modularized one through OSGi has enabled OSC to implement these goals.

Bringing together a diverse group of security services and software defined network (SDN) providers was key to building the OSC community. This was possible due to the OSC plugin model using OSGi which allows specific vendors to integrate OSC with their existing solutions with minimum effort. Beyond its plugin model OSC also makes use of OSGi in its core components. This enables better extensibility, testability and overall maintainability of its code base. These aspects are key to OSC as it relates to the new and dynamic areas of SDN and security VNFs in cloud and data center environments and it must be able to easily adapt to the constantly changing requirements and technologies pertaining these fields.

In this talk we will explore the journey for turning OSC into a modularized application using OSGi, highlighting some of the achieved results, key learned lessons and the next steps for this work.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Journey from Monolith to a Modularized Application - Approach and Key Learnings - E Xavier & T Ward

  1. 1. AJourneyfromaMonolithtoa ModularizedApplication ApproachandKeyLearnings Emanoel Xavier Senior Software Developer, Intel Tim Ward CTO, Paremus 1
  2. 2. 2 Notices and Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit Intel, the Intel logo and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2017 Intel Corporation.
  3. 3. Modularity through the ages… 3 A little history
  4. 4. 4 Adam Smith The Wealth of Nations (1776) Division of Labor “…the trade of a pin-maker: a workman not educated to this business could scarce, perhaps, with his utmost industry, make one pin in a day, and certainly could not make twenty.” “But in the way in which this business is now carried on, it is divided into … about eighteen distinct operations, which, in some manufactories, are all performed by distinct hands” “Those ten persons, therefore, could make among them upwards of forty-eight thousand pins in a day. Each person, therefore, making a tenth part of forty-eight thousand pins, might be considered as making four thousand eight hundred pins in a day.” SMALLER SPECIALIZED CONCURRENT
  5. 5. 5 Henry Ford The Moving Assembly Line (1910) Modularity & Productivity parts and car “come to you” 83 parts. 1 step/min from 12h to 90 mins “continuous build” interchangeable parts model
  6. 6. “The major advancement in the area of modular programming has been the development of coding techniques and assemblers which (l) allow one module to be written with little knowledge of the code in another module, and (2) allow modules to be reassembled and replaced without reassembly of the whole system. This facility is extremely valuable for the production of large pieces of code, but the systems most often used as examples of problem systems are highly- modularized programs and make use of the techniques mentioned above.” – David Parnas, On the Criteria to Be Used in Decomposing Systems into Modules (1972) 6 Modularity in Software
  7. 7. 7 Apollo 16 Lunar Lander Module (1972) “... the systems most often used as examples of problem systems are highly-modularized programs and make use of the techniques mentioned above.” A telling phrase… “Modularity is too hard to be worth it” Mixing the problem with the solution The problems are complex Modularity makes the solution possible
  8. 8. Modularity today 8 Fast forwarding
  9. 9. 9 1970s 2017 Modern modular software – vision
  10. 10. 10 1970s 2017 Modern modular software – reality
  11. 11. 11 Modularity is not magic Forgotten past lessons 70s: Low level modularity for low level coupling Late modularity == expensive tech debt
  12. 12. (Serves One) 12 The recipe for a monolith
  13. 13. Starting a new project 13 Nobody plans to write bad code Things will be different now Well defined, modular design we have all been there Sonar*, Findbugs*, 100% CC intended modularization
  14. 14. Growing the project 14 Mgmt needs more code Here, more devs Flexible processes more requirements. 6 months deadline but, Karl, your best coder is gone That is ok, we are “Agile*”
  15. 15. Growing the project (6 months later) 15 almost done processes and tools some workarounds needed worked overtime. “paper” release people catching up with Karl’s code. code reviews that’s ok. track and fix later
  16. 16. Renewed interest (6-12 months later) 16 customer: shows interest sales: delivery in 6 months no one is familiar with the code +4 new members, +20 years worth work +1 headache no previous team member. Karl’s code is still around
  17. 17. 17 Do any parts of this seem familiar? Gall’s Law: “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”  Nobody sets out to write bad code  Successful projects are bound to become complex  Complexity must be managed
  18. 18. (Serves One) 18 How OSGi helps
  19. 19. 19 imposes API boundaries hides implementation details automates dependency management enforces dependencies at runtime facilitates composition
  20. 20. 20  Won’t magically turn a monolith into a modularized application  Will provide you the tools for that  Won’t automatically fix all coupling issues, “shortcuts” and technical debt  Will make them more evident  Won’t fix API leaks  Will enforce your APIs
  21. 21. 21 Getting results Planned and managed API changes Work with it, not against it OSGi: ongoing commitment
  22. 22. How OSGi helps 22  Limits complexity to smaller pieces – Divide and conquer  Pieces can be more easily and predictably replaced – No runtime surprises  Easier to understand dependencies fan out  Better testability
  23. 23. 23 OSC 101
  24. 24. 24 How it started  Intel closed source project  Application and deployment of security virtual functions  Let’s extend it !  Let’s build a community around it! – Spring 2016 => Summer 2017 Seemsfamiliar?
  25. 25. 25 OSC in SDI LOCATIONUTILIZATION LATENCY SECURITY Infrastructure Attributes POWER PERFORMANCE Infrastructure Attributes Servers Virtualization Containers Virtual Router Virtual SAN Scale Out Storage Apps SDN/NFV SDS Infrastructure Pool VMs Virtual Firewall Virtual Switch Policy based automation Self service portal Service Orchestration Compute Network Storage Security Pool of virtualized resources Platform capability aware Centralized and automated
  26. 26. register setup 26 OSC in action Plugins OSC Security Manager SDN Controller Data Center  Plugins  VNF Image  Virtualization Connector  Manager Connector  Security Group  Deployment  Security Policies IPS, NGFW, ADC
  27. 27. 27 OSC logical components osc-core sec-mgr-plugins sdn-ctrlr-plugins sdn-ctrlr-apisec-mgr-api OpenStack4j* RabbitMq* Fabric8* OpenStack KubernetesH2 DB VNF Catalog Plugins BL services Sync tasks Security Mgr Services SDN Ctrlr Services UI REST API OSGi , h2db-jpa, apache.aries.jpa, vaadin, logback, …
  28. 28. 28 The path to open source Community members diversity SDN Providers and Security Managers neutral ! OSC Plugins Model Dependency conflicts REST Services?  Redundant endpoints  Not conducive to early adoption  Legacy plugins
  29. 29. 29 The journey starts…
  30. 30. 30 Modularizing the plugins osc-server sec-mgr-plugins 1. Build dependencies 2. Runtime dependencies 3. Startup dependencies osc-controller sdn-ctrlr-plugins sec-mgr-apisec-mgr-api pom.xml pom.xml whiteboard pattern whiteboard pattern OSGi Resolver BAR Installer
  31. 31. 31 Modularizing the plugins osc-server sec-mgr-plugins osc-controller sdn-ctrlr-plugins sec-mgr-apisec-mgr-api whiteboard pattern whiteboard pattern OSGi Resolver BAR Installer  Smaller footprint  Simpler service registration  Earlier plugin validation 64 /110 packaged dependencies OSGi Http Service Whiteboard Plugins Lifecycle
  32. 32. 32 Are we modular yet?
  33. 33. 33 Perceived modularization Implemented Lacking "…Curved Walking in Humans” Gregoire Courtine, Journal of Neurophysiology
  34. 34. 34 Does it matter? SDN/SDI maintainability clean dependencies semantic versioning isolation extensibility Open Source incremental build clean dependencies semantic versioning runtime enforcement unit testing
  35. 35. 35 The journey continues….
  36. 36. 36 OSC uber bundle anatomy  Common monolithic to modularized migration path  Flat classpath  Transparent non-bundle dependencies  No isolation between internal packages osc-server uber bundle uber bundle core BL services Sync tasks DB access UI REST API
  37. 37. 37 Database access: osc-domain  Only @Entity classes  JPA API & Apache Aries JPA  OSGi Transaction Control Service  Unidirectional dependency osc-server uber bundle core BL services Sync tasks DB access UI REST API osc-domain hibernate-core, h2databasehibernate-jpa-api
  38. 38. 38 User interface: osc-ui  New project: osc-service-api  UI views as DS components  OSGi managed views osc-server uber bundle core BL services Sync tasks UI REST API osc-domain osc-ui osc-service-api
  39. 39. 39 REST API: osc-rest-server  Uses the osc-service-api  JAX-RS service classes  No more domain entity references from the DTOs  New service APIs and DTOs osc-server uber bundle core BL services Sync tasks REST API osc-domain osc-ui osc-service-api osc-rest-server
  40. 40. 40 Modularized OSC  5 new smaller bundles  DS, Interfaces, clear contracts  Improved dependencies scopes  Auxiliary uber bundles – Non OSGi dependencies  Incremental build osc-server osc-server core BL services Sync tasks osc-domain osc-ui osc-service-api osc-rest-server +
  41. 41. 41 Are we modular yet? End of journey?
  42. 42. No… 42  Modularization, like good code, is an ongoing effort  There are multiple degrees of modularization. – A gradual approach is recommended – But don’t go back.  OSGi is not a magic wand, but makes the job possible A monolith will only get you so far. For a longer, pleasant (and modular) journey you will need not only the right tools and frameworks. The right mindset is just as important.
  43. 43. Thankyou! Emanoel Xavier Senior Software Developer, Intel Tim Ward CTO, Paremus 43 Come join our community!