Software development effort reduction with Co-op

897 views
795 views

Published on

This talks explains the motivations for the Co-op technology: what are the challenges it addresses, in particular focusing on reducing accidental complexity, where it comes from, and a general vision on how to resolve it. Then we continue to show practical application of Co-op, including experience figures from large-scale application of a previous generation of this technology. Show a little bit about its realization, and conclude with an evaluation of the technology.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
897
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Venture Lab panel presentation Lodewijk Bergmans
  • RoI factors: 400-4000% 70-1000% 150-700% -500% (c) 2010 steX bv – www.stexbv.com
  • quote: “7 th . Law of Computer programming: Program complexity grows until it exceeds the capability of the programmer who must maintain it.” 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • Many composition mechanisms... includes domain-specific ones such as exemplified by many design patterns! Note that these are all *well-motivated* with (specific) examples where the specific technique is superior to other ones... so, why so many??  5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • Inheritance vs delegation static vs dynamic, sharing structure vs sharing state Smalltalk inheritance vs Beta inheritance robustness/predictability vs. flexibility/extensibility inheritance & aggregation vs. aspects local reasoning vs. global reasoning & maintainability so what do ‘real engineers’ do?  5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • if you look at the construction of houses, cars or even much smaller devices, a lot of different construction techniques are combined because each incurs trade-off: easy-to-use, fast, cheap, strong, can be undone, .. 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • just a handful! it appears as if only nails & rope are sufficient to build the complex systems that we do (AspectJ is a notable exception) typically these include message invocations, object aggregation and inheritance with specific (always fixed) semantics we need to address this issue  5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • Many composition mechanisms... includes domain-specific ones such as exemplified by many design patterns! Note that these are all *well-motivated* with (specific) examples where the specific technique is superior to other ones... so, why so many??  5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • Time is ready for designing languages without fixed & built-in composition mechanisms: Hypothesis: Prog. languages of the (some) future will support extensible/tailorable composition mechanisms. 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • old news: even Darwin said it 1,5 century ago. According to Darwin in “the origin of species” In fact it is not an exact quote, but a paraphrase of –probably- Megginson in 1963 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • -- And later, about Fortress, in “a growable language”: “ The result is that we have a fairly complicated language for library writers that enables them to write libraries that present a relatively simple set of interfaces to the application programmer.” and in Guy Steele talks about language design in the January Doctor Dobb's. “ I'm not saying we should throw structured programming out the window. I am saying that the trade-offs have changed and are likely to keep changing.” 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • Piumarte & Warth “Open extensible object models” in Hirschfeld, rose (eds) Self-sustainable systems (LNCS) 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • composition mechanisms must be composable within the same application also on same abstractions depending on semantics! (inherent incompatabilities) e.g. combining beta- and smalltalk inheritance in same hierarchy makes no sense. closure: composition mechanisms are first-class citizens -> scalable model 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • show a before and after code view!! wher eit is used show/flash the Observe co-operator code?o rewrite in Magik syntax?? Venture lab panel presentation: background material 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute
  • multiple inheritance (reuses single inheritance) tracing abstraction (with & without PA reuse) 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • not all instances are shown! 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • It is about a suitable abstraction level for representing composition operators; this shows the outline of our abstraction. we explain it by showing the computation model/ execution semantics (in fact we believe that in an ever more dynamic world, it is important to have a computation model, so things can be made dynamic, whenever required) 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • evaluate binding: - evaluate constraints to determine a partial ordering and evaluate the first match(es) binding of context variables between incoming and outgoing event 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • dynamic languages flexible, expressive compositions DSLs (also: polyglot & multi-paradigm programming) right abstractions for right job Model Driven Engineering separate (implementation details) and logic, avoid boilerplate code meta-modelling & meta-programming allow for separation of concerns, modularizing common solutions in a transparent manner conclusions: we believe dynamic languages fit more natural wih a computation model, rather than a transformational approach. choose the appropriate language abstractions for expressing a program where possible design domain-specific abstractions once and reuse we are more used to jumping up and down meta-levels and handling higher-level abstractions today 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • e.g. CLOS: the notion of parents is fixed in the design of the MOP 5/20/10 (c) 2009-2010 stex BV, confidential: please do not distribute Venture lab panel presentation: background material
  • Software development effort reduction with Co-op

    1. 1. Software development effort reduction with Co-op Dr.ir. Lodewijk Bergmans [lbergmans@acm.org]
    2. 2. About me <ul><li>software composition </li></ul><ul><ul><li>Objects </li></ul></ul><ul><ul><li>Aspects </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><li>Industry-as-a-laboratory </li></ul><ul><ul><li>ASML, Oce, Siemens, .. </li></ul></ul><ul><li>European Network of Excellence on AOSD </li></ul><ul><li>Assistant Prof., University of Twente </li></ul><ul><li>Independent consultant (1994-1997, 2008-) </li></ul><ul><ul><li>teach & mentor SE </li></ul></ul><ul><ul><li>software architecture </li></ul></ul><ul><li>e.g. Philips Medical Systems, Ernst & Young MC, Ordina Panfox </li></ul><ul><li>Ericsson Mobile Communications (Lund) </li></ul><ul><li>Independent consultant, steX </li></ul><ul><li>Research </li></ul><ul><li>Industry </li></ul>
    3. 3. Message of this talk <ul><li>Development and evolution cost are high: </li></ul><ul><ul><li>for frameworks, applications & customizations </li></ul></ul><ul><li>Cost can be reduced by: </li></ul><ul><ul><li>reducing accidental complexity </li></ul></ul><ul><ul><li>improved standardization </li></ul></ul><ul><ul><li>easier reuse of design solutions (idioms & patterns) </li></ul></ul><ul><li>Co-op is an enabling technology </li></ul>
    4. 4. Contents <ul><li>Challenges in software development </li></ul><ul><li>Understanding the problems </li></ul><ul><li>A solution approach  Co-op </li></ul><ul><li>Application of the Co-op solution technology </li></ul><ul><li>Practical applicability: previous experiences </li></ul><ul><li>Technology benefits </li></ul>
    5. 5. Challenges in software development
    6. 6. Some major issues in SE <ul><li>managing complexity </li></ul><ul><ul><li>we build increasingly (inherently) complex systems </li></ul></ul><ul><ul><li>which carry more and more accidental complexity </li></ul></ul><ul><li>continuous change </li></ul><ul><ul><li>change starts on day one.. </li></ul></ul><ul><ul><li>causes large maintenance costs (60%-90%) </li></ul></ul>
    7. 7. Experience: complexity vs project risk source: Wallace, Keil & Rai, “Understanding software project risk: a cluster analysis”, Information & Management 42 (2004)
    8. 8. Improvement potential source: Walker Royce, “Improving Software Economics”: IBM white paper (2009) costs (per person year) 10-35% 5-10% <5% 200-1000% 25-100% 15-35% 5-25% impact (productivity) 25-50%
    9. 9. Technical challenges <ul><li>modularization of software </li></ul><ul><ul><li> avoid accidental complexity </li></ul></ul><ul><ul><li>but how to keep concerns clean and well-separated </li></ul></ul><ul><ul><ul><li>How to retain the design within the actual implementation </li></ul></ul></ul><ul><li>architectual integrity </li></ul><ul><ul><li>Does implementation conform to the architecture? </li></ul></ul><ul><li>optimal usage of skills </li></ul><ul><ul><li>advanced solutions may not be effective for all developers... </li></ul></ul><ul><ul><li>Can we encapsulate design solutions and make them easy to (re-)use ? </li></ul></ul>
    10. 10. <ul><li>Where does accidental complexity come from? </li></ul>Understanding the problem
    11. 11. Key technique in SE: <ul><li>divide and conquer: </li></ul><ul><ul><li>decompose into modules </li></ul></ul><ul><ul><li>that are again composed into a system </li></ul></ul><ul><li>hence modules/abstractions and corresponding composition techniques: </li></ul><ul><ul><li>are a key element in computer science </li></ul></ul><ul><ul><li>mark each paradigm shift in the history of sw.dev. </li></ul></ul><ul><ul><li>continously new proposals appear... </li></ul></ul>A composition technique (‘composition operator’) combines the behavior of two or more program elements.
    12. 12. Result: subroutine co-routine aggregation inheritance (Smalltalk style) inheritance (C++ style) inheritance (BETA style) multiple inheritance dynamic inheritance associative inheritance delegation predicate dispatch mixin inheritance traits point-cut-advice composition inter-type declarations roles contracts composition filters dependency injection parameterization actors hyperspaces layered object model
    13. 13. <ul><li>Why? Each technique involves trade-offs </li></ul><ul><li>e.g. Smalltalk  C++ inheritance </li></ul>
    14. 14. Composition techniques in the physical world
    15. 15. <ul><li>very few </li></ul><ul><li>fixed by language </li></ul><ul><li>thus often not the best trade-off! </li></ul><ul><li>Composition techniques in software systems </li></ul>
    16. 16. Result: accidental complexity <ul><li>c.f. extra glue, nails & screws: </li></ul><ul><ul><li>ugly and hard to attach, understand, and modify </li></ul></ul><ul><li>because the composition in the problem domain cannot be expressed adequately in the implementation: </li></ul><ul><ul><li>extra ‘glue’ code to implement relations </li></ul></ul><ul><ul><li>cannot properly separate concerns </li></ul></ul><ul><ul><ul><li>Or need additional design patterns ->more code & complexity </li></ul></ul></ul><ul><ul><li> much harder to maintain! </li></ul></ul>
    17. 17. The solution approach
    18. 18. programming languages should not fix composition mechanisms, but make them extensible and tailorable.
    19. 19. “ it is not the strongest of the species that will survive, or the most intelligent. It is the most adaptable to change.” 1859 Darwin (Megginson)
    20. 20. “ We need to put tools for language growth in the hands of the users.” 1998 Steele
    21. 21. “ ... lets [the programmers] [..] express concise solutions and free the original language designer from ever having to say &quot;I'm sorry&quot; ” 2008 Piumarte & Warth
    22. 22. Co-op (‘ Co mposition -Op erator’) solution <ul><li>language enhancement that: </li></ul><ul><li>allows defining wide range of compositions </li></ul><ul><ul><li>including domain-specific ones </li></ul></ul><ul><ul><li>including solution patterns </li></ul></ul><ul><li>compositions are composable </li></ul><ul><li>compositions are first-class citizens </li></ul><ul><ul><li>so they can be part of the solution domain </li></ul></ul><ul><ul><li>to support reuse, extension and adaptation (scalability) </li></ul></ul>
    23. 23. <ul><li>or: how application developers can easily (re)use architectural strategies and tactics </li></ul>Technology Application
    24. 24. Usage of Co-Op <ul><li>Compositions are first-class abstractions </li></ul><ul><ul><li>provided by standard library, or </li></ul></ul><ul><ul><li>custom developed by third party, or </li></ul></ul><ul><ul><li>custom developed within organisation </li></ul></ul><ul><li>Using them is comparable to instantiating and initializing objects.. </li></ul><ul><ul><li>CompOperator1 newBetween: objectA and: objectB; </li></ul></ul><ul><ul><li>CompOperator2 newBetween: ClassA and: ClassB; </li></ul></ul><ul><ul><li>CompOperator3 newBetween: objectA and: objectB and: objectC; </li></ul></ul><ul><ul><li>... </li></ul></ul>
    25. 25. Example of using Co-Op Patterns: Observer Pattern <ul><li>Library provides simple Observe composition: </li></ul><ul><li>Creates a new observer (action) that is to be invoked whenever the methods show and hide have executed on object window: </li></ul><ul><li>Observe newSubject: window messages: ( ((List new) add: &quot;show”) add: &quot;hide”) observerAction: [ Console write: &quot;!!! window visibility = “; Console writeln: (window isVisible) ] ; </li></ul>
    26. 26. A quick view on the definition of the operator <ul><li>module Observe { </li></ul><ul><li>var @observerBehavior; </li></ul><ul><li>init(subject, listOfOperations, observerBehavior) { </li></ul><ul><li>var callsToSubject; </li></ul><ul><li>var sendToObserver; </li></ul><ul><li>var binding; </li></ul><ul><li>var constraint; </li></ul><ul><li>@observerBehavior = observerBehavior; </li></ul><ul><li>callsToSubject = [Selector new: </li></ul><ul><li> { return [[[event target] isSameObject: subject] and: [listOfOperations contains: [event selector]]]; } ]; </li></ul><ul><li>sendToObserver = [Selector new: { return [OperationRef new: &quot;Observe&quot;, &quot;notify&quot;, &quot;inner&quot;, this]; } ]; </li></ul><ul><li>binding = [Binding new: callsToSubject, sendToObserver, { return [Dictionary new]; }]; </li></ul><ul><li>constraint = [PreConstraint new: defaultCallBinding, binding]; </li></ul><ul><li>[constraint activate]; </li></ul><ul><li>[binding activate]; </li></ul><ul><li>} </li></ul><ul><li>notify() { </li></ul><ul><li>[Console writeln: &quot;this=&quot;, this]; </li></ul><ul><li>[@observerBehavior execute]; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    27. 27. Example of using Co-Op Patterns: State Pattern <ul><li>fsm = StatePattern newWithContext: this initState: closedState; </li></ul><ul><li>fsm addTransitionFrom: closedState action: &quot;openPort&quot; to: listenState; </li></ul><ul><li>fsm addTransitionFrom: listenState action: &quot;receiveSyn&quot; to: synReceivedState; </li></ul><ul><li>... </li></ul>
    28. 28. Summary <ul><li>This illustrates how design level solutions </li></ul><ul><ul><li>typically involving multiple modules </li></ul></ul><ul><li>can be defined and standardized once, </li></ul><ul><li>And can be instantiated directly by application developers </li></ul><ul><ul><li>in the language that they know </li></ul></ul><ul><ul><li>without needing to know the complexity behind those solutions </li></ul></ul><ul><ul><li>In other words: they become empowered to use advanced techniques </li></ul></ul>
    29. 29. Practical applicability
    30. 30. Large-scale industrial case <ul><ul><li>4 concepts ‘crosscut’ the system (-30% of code) </li></ul></ul><ul><ul><li>error-prone, hard to maintain, ... </li></ul></ul>int get_kng(KNG_struct* KNG_ptr) { const char* func_name = &quot;get_kng&quot;; int result = OK; timing_handle timing_hdl = NULL; TIMING_IN; trace_in(mod_data.tr_handle, func_name); if (result == OK) { /* Retrieve current KNG */ *KNG_ptr = mod_data.KNG; } HE(result, &quot;GET_KNG FAILED&quot;); trace_out(mod_data.tr_handle, func_name, result); TIMING_OUT; return result; } primary functionality error handling
    31. 31. Industrial case: approach <ul><li>capture the essence of the 4 concepts </li></ul><ul><ul><li>separate these into independent abstractions </li></ul></ul>KNG_struct get_kng() { return ( mod_data.KNG); }
    32. 32. Industrial case: numbers <ul><li>We did a test migration for a sample module </li></ul><ul><ul><li>on three aspects: </li></ul></ul><ul><ul><ul><li>code reduction of over 20% for a sample module </li></ul></ul></ul><ul><li>Company estimated potential benefits (COCOMO): </li></ul><ul><ul><li>7-10% effort reduction on software </li></ul></ul><ul><ul><li>3% lead time reduction on software </li></ul></ul><ul><li>Later controlled experiments for the tracing aspect : </li></ul><ul><ul><li>initial programming effort reduction: 6% </li></ul></ul><ul><ul><li>(severity of) error reduction: 77% </li></ul></ul>
    33. 33. Technology realization
    34. 34. Co-op Proof-of-concept <ul><li>Based on well-structured event reflection </li></ul><ul><li>Implemented on Java </li></ul><ul><ul><li>with Eclipse integration (including debugger)  </li></ul></ul><ul><li>And two prototypes in Miranda resp. Haskell </li></ul><ul><li>Library with sample composition operators: </li></ul><ul><ul><li>single inheritance (3) </li></ul></ul><ul><ul><ul><li>BETA inheritance </li></ul></ul></ul><ul><ul><li>multiple inheritance </li></ul></ul><ul><ul><li>delegation (2) </li></ul></ul><ul><ul><li>point-cut – advice (2) </li></ul></ul><ul><ul><li>traits (cf. mix-ins) </li></ul></ul><ul><li>tracing – abstraction </li></ul><ul><li>memoization – abstraction </li></ul><ul><li>subject – observer </li></ul><ul><li>state – abstraction </li></ul>
    35. 35. IDE integration sample
    36. 36. Execution model perspective
    37. 37. Composition Model primitives <ul><li>Module </li></ul><ul><li>Event </li></ul><ul><ul><li>properties: event name, sender, target, lookup type, event annotation, parameters </li></ul></ul><ul><li>Module specifications (may) include: </li></ul><ul><ul><li>Binding: </li></ul></ul><ul><ul><ul><li>Event Selector </li></ul></ul></ul><ul><ul><ul><li>Action Selector </li></ul></ul></ul><ul><ul><ul><li>Context Mapping of events </li></ul></ul></ul><ul><ul><li>Constraints </li></ul></ul><ul><ul><ul><li>between bindings </li></ul></ul></ul>
    38. 38. A helicopter view <ul><li>The event dispatch process </li></ul>filter bindings match bindings apply constraints & evaluate bindings
    39. 39. Trends in sw. development <ul><li>Upcoming technologies: </li></ul><ul><ul><li>dynamic languages </li></ul></ul><ul><ul><li>DSLs (also: polyglot & multi-paradigm programming) </li></ul></ul><ul><ul><li>Model-Driven Engineering </li></ul></ul><ul><ul><li>meta-modelling & meta-programming </li></ul></ul><ul><li>Common theme: separation of concerns </li></ul><ul><ul><li>(re-)use of tailored (application specific) abstractions that can be parameterized </li></ul></ul><ul><ul><li>avoid boilerplate code (generate) </li></ul></ul>
    40. 40. About related work <ul><li>LOTS! </li></ul><ul><li>several frameworks that support variety of composition operators. </li></ul><ul><ul><li>few that unify aspect and object composition in one mechanism. </li></ul></ul><ul><li>Several approaches for extensible languages </li></ul><ul><ul><li>we focus on extending with new composition operators (only) </li></ul></ul><ul><li>MOPs: </li></ul><ul><ul><li>our approach is compatible with a MOP approach </li></ul></ul><ul><ul><li>different from existing MOPs: </li></ul></ul><ul><ul><ul><li>no assumptions about (fixed) composition operators </li></ul></ul></ul><ul><ul><ul><li>unify aspect & object composition mechanisms </li></ul></ul></ul><ul><ul><ul><li>allow composition of composition operators </li></ul></ul></ul>
    41. 41. Evaluation
    42. 42. Benefits per stakeholder audience benefits application developer - straightforward application of canned solutions - write less code (=> less bugs) software designer - designs are explicit in code, and localized - more robust to change - can choose optimal solutions (instead of a few fixed ones) senior SE/sw. architect - more opportunities to (guarantee) use of standard solutions - more opportunities to keep code and design consistent - better tools to manage complexity Sw. manager/project leader - better consistency (i.e. quality) - managing complexity  more control over project - empower all team members to build better software easier. end-user/ product owner - software that has fewer problems - software that is flexible: easier/cheaper to maintain and evolve
    43. 43. Co-op: an agile enabler <ul><li>value code over documentation: </li></ul><ul><ul><li>design patterns & standards can be expressed in software </li></ul></ul><ul><ul><li>everything is expressed in same language </li></ul></ul><ul><li>simplicity: </li></ul><ul><ul><li>co-op helps to keep the software simple </li></ul></ul><ul><ul><li>co-op helps to reduce the amount of code </li></ul></ul><ul><li>Embrace change </li></ul><ul><ul><li>better designs and code that matches the design enables fluent change </li></ul></ul>
    44. 44. Conclusion: Co-op application <ul><li>Managing complexity of large software </li></ul><ul><ul><li>technology is a good match for agile development </li></ul></ul><ul><li>Offering better means for reuse and evolution </li></ul><ul><ul><li>helps with customization and version issues </li></ul></ul><ul><li>Common solutions can be </li></ul><ul><ul><li>standardized, reused and/or enforced </li></ul></ul><ul><ul><li>can lead to substantial code reductions </li></ul></ul><ul><ul><li> better software quality </li></ul></ul><ul><ul><li> empowering application programmers </li></ul></ul>
    45. 45. Thanks for your attention <ul><li>questions? </li></ul><ul><li>Suggestions? </li></ul>assistant prof., SE groep / EEM CS University of Twente, The Netherlands email: lbergmans@acm.org phone: +31-53-4894271 web: trese.cs.utwente.nl/~bergmans software engineering specialist steX bv, The Netherlands email: lodewijk@stexbv.com cell phone: +31-651100838 web: www.stexbv.com Dr.ir. Lodewijk Bergmans

    ×