From agile development to agile evolution of enterprise systems


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • With a Ph.D. in information technology (IT), I have always worked in the provision of IT services. The posts that I have occupied have progressed through the phases of programmer, developer, business analyst, project manager to, most recently, enterprise solutions architect . Over the last several years, I successfully architected and delivered several IT solutions for different needs ranging from a heavy-duty production process to a federated distributed organization. I have innovated and employed an architectural framework for improving business process management systems. This framework is agile, service-oriented and provides for the integration of people, processes, systems and IT tools. Aiming towards achieving a good synergy between business needs and IT potentials, I have always managed to earn the respect and appreciation of users and clients. Skills and Experience Analysis and conceptual design of service-oriented architectures . A leadership role in the design and implementation of the automation of production business processes Experience in the design, implementation and maintenance of IT systems Successful use of the agile approach to software development . Strong leadership in the management of informal groups. Extensive experience with IT technologies Last position: " Chief Enterprise Solutions Architect " responsible for the architectural design, implementation, deployment and evolution of enterprise systems. The client base comprised international organizations, multinational and local companies, as well as public sector services in the French speaking part of Switzerland. Some work done: - development a ECM-based architecture for a private bank in Geneva (mainly for systems integration and business process automation), - demonstration of the Web Content Management solution for a public hospital, - preparation a proposal based on ECM products for a cantonal library, - implementing a working prototype of the document management for the ISO 9001 quality management system for an SME in Lausanne, - installation and evaluation of tools in a scope of a short-term contract with an international organization in Geneva for a pilot project on the archiving e-mails, and - discussing with several clients how ECM products and the “right” architecture can be used to transform their business operations to improve significantly their effectiveness and efficiency.
  • The illustration is taken from the book “The innovator’s dilemma” Plone is a disruptive technological innovation Plone already delivers performance demanded at the low end of the market Assuming that Plone targets also the high end of the market Treat Plone as an “adult” on the ECM and BPM market and in the enterprise environment
  • To automate and manage typical business procedures a multi-layer model has been used. Each layer is a level of abstraction of the business and addresses some particular concerns: The business data layer comprises information that is stored in existing repositories and traditional applications. The business objects layer comprises objects (actually, business data containers) specific for a particular business. The business regulations (and rules) layer comprises the actions on the business objects, which must be carried out to perform the business tasks. The business execution layer carries out the business procedures (each procedure is an orchestrated set of manual and automated tasks). The business modelling (and monitoring) layer analyses the business events, which summarize execution of the business procedures. The business intelligence layer implements enterprise-wide planning, performance evaluation and control actions applied to the business procedures. Each layer has some responsibilities; it uses the lower layer functionality, and it serves the higher layer. Each layer has a well-defined interface and its implementation is independent of that of others. Each layer comprises many building blocks that could be reused in different activities. Another practical observation is that different layers have life-cycles of different time scales: typical repositories have a 5-10 year life-span while the business requires continuous improvement. Because of the implementation independence of different layers, each layer may evolve at its own pace without being hampered by others. As a rule, the existing applications already implement many of business objects, relegations and events, but in a much unstructured way. The lasts are intermixed, not reusable and to be implemented again and again. This model is a tool which helps the organization to design system in business terms, but not in IT tools. Also, this model doesn't mandate to have all layers at the same time and to provide them in one project.
  • This model is applied for both meta- and micro-projects. In an established architectural framework, we don’t actively involved into micro-projects. (if system works without you then you did a good job). But, it is critical to control evolution of the architectural framework itself.
  • As the framework changes project management, two types of projects emerge; micro-projects for agile implementations of new features and meta-projects for framework maintenance and development. The latter resembles the Deming wheel: "plan" the feature(s) to be accomplished next as a micro-project according to results measurement and business priorities; "carry out" the changes; "validate" the changes; "analyse" the effect of the changes on the system and eventually propose unifications if, for example, a local modification was effective and should therefore be used to a wider extent. With a typical time per cycle per micro-project of 2 to 3 weeks, such project management is inherently different from the traditional multi year planning and the whole process becomes similar to maintenance rather than development. The descriptive below shows micro-projects mapped into phases of traditional IT projects. A note about timing – all values are real. They were achieved for automation of a production environment. Typical duration of a micro-project should correspond to a typical duration of related business. Typical timing of micro-projects: Definition phase: less than an hour. Specification / conception phases: one day Development / test / validation phases: from hours to days (depending on user’s availability). Deployment phase: practically instant. Definition phase: Business optimisation potential is evaluated. Features to be implemented are understood. Priorities and availability of features are communicated. Specification / conception phases: A product prototype using a workflow, an electronic form or a screen dump is used to validate what will be implemented. Missing components (if any) are identified and specified. Missing components (if any) are evaluated; use of new tools and/or utilities are justified. Development / test / validation phases: Missing components (if any) are incrementally developed, tested and validated. Complete product is assembled from existing and new components. The product is incrementally tested, validated and deployed. Extra monitoring, where necessary, is deployed. Resources requirements are estimated and the system is reconfigured to provide them. Production phase: The new product or latest version of the product is made available for use. In case of product or version replacement, verification of previous product retention cycle The documentation issue is addressed and simplified in several ways: Built-in document quality management system. Maximal use of visual tools (e.g. workflow, forms). User modifications are documented by the users IT-specific documentation is largely derived from the meta-project(s) and their adopted patterns. Programs, such as workflows, e-forms and schemas, work as documents and vice-versa.
  • From agile development to agile evolution of enterprise systems

    1. 1. From agile development to agile evolution of enterprise systems Dr Alexander Samarin Clio S.A. EuroPython conference 2006, CERN, Geneva, Switzerland
    2. 2. Who am I? An enterprise solutions architect <ul><li>Have always worked in the provision of IT services </li></ul><ul><li>From a programmer to a systems architect </li></ul><ul><li>Experience in academic, international and industry environments: CERN, ISO, IOC, BUPA </li></ul><ul><li>Have created systems which work without me </li></ul><ul><li>Current specialisation is improving business process management systems </li></ul><ul><ul><li>effectiveness (“Do the right things”) </li></ul></ul><ul><ul><li>efficiency (“Do the things right”) </li></ul></ul>
    3. 3. Agile software development is a classic case of disruptive technology <ul><li>The customers appreciate it </li></ul><ul><ul><li>an order of magnitude increase in IT value for the customers </li></ul></ul><ul><ul><li>no comprehensive up-front specs </li></ul></ul><ul><ul><li>results-oriented </li></ul></ul><ul><li>The IT establishment criticises it </li></ul><ul><ul><li>bad or no design </li></ul></ul><ul><ul><li>no documentation </li></ul></ul><ul><ul><li>works only for top professionals </li></ul></ul>
    4. 4. My position: agile means adaptable <ul><li>Agility is the ability not only to create change but to respond to change </li></ul><ul><li>Agility is the ability to balance flexibility and structure </li></ul><ul><li>Gartner: Agility is the ability of an organization to sense environmental change and to respond to it efficiently and effectively </li></ul>
    5. 5. A daunting optimisation task <ul><li>From a typical enterprise environment: </li></ul><ul><ul><li>a complex system of systems that has grown over years </li></ul></ul><ul><ul><li>a very hostile environment for new things </li></ul></ul><ul><li>To a flexible business system which is easily adaptable to: </li></ul><ul><ul><li>policies, priorities, existing data, IT systems, business processes, size, complexity, budgets, culture, etc. </li></ul></ul><ul><li>Subject to socio-technical aspects: </li></ul><ul><ul><li>how you do something may be more important than what you do </li></ul></ul>
    6. 6. Lean production is an example of optimisation in industry <ul><li>See the whole picture </li></ul><ul><li>Learn constantly </li></ul><ul><li>Decide as late as possible </li></ul><ul><li>Deliver as fast as possible </li></ul><ul><li>Eliminate waste </li></ul><ul><li>Empower the team </li></ul><ul><li>Build in integrity </li></ul><ul><li>Avoid sub-optimisation </li></ul>
    7. 7. Harvard Business School studies: development practices that spell success <ul><li>Early release of the evolving product design to customers </li></ul><ul><li>Daily incorporation of new software code and rapid feedback on design changes </li></ul><ul><li>Teams with broad-based experience of shipping multiple projects </li></ul><ul><li>Major investments in the design of the product architecture </li></ul>
    8. 8. A dilemma <ul><li>Agile software development is sound </li></ul><ul><li>… although there are valid criticisms (agile development is like racing a yacht and is not necessarily suited to captaining a liner) </li></ul><ul><li>We need it to deliver agile evolution of enterprise systems </li></ul><ul><li>Heuristic: sometimes it is necessary to expand the concept in order to simplify the problem </li></ul>
    9. 9. The main lesson from agile development: see and understand the big picture
    10. 10. Critical aspects for agile evolution of enterprise systems <ul><li>The big picture (i.e. the enterprise architecture) is </li></ul><ul><ul><li>available </li></ul></ul><ul><ul><li>understood </li></ul></ul><ul><ul><li>agreed internally by consensus </li></ul></ul><ul><ul><li>not “parachuted” by consultants or a vendor </li></ul></ul><ul><ul><li>addressing Business Process Management (BPM) </li></ul></ul><ul><li>Development of the architecture for a particular enterprise should not be a project that takes many man-years and reams of pages </li></ul><ul><li>Business and IT use common tools </li></ul>
    11. 11. Enterprise means Business Process Management <ul><li>Business Process Management (BPM) allows you to model , automate , control , measure and optimise the flow of business process steps </li></ul><ul><li>It spans your organisation’s systems, people, customers and partners within and beyond your corporate boundaries </li></ul><ul><li>Gartner estimates that there are currently over 140 business process management vendors </li></ul>
    12. 12. My approach to BPM (providing a fishing rod, not a fish) <ul><li>Any enterprise has its own BPM system; some enterprises want to change it </li></ul><ul><li>An offer of an architectural framework for improving BPM systems </li></ul><ul><li>It makes use of the synergy that exists between business needs and IT potentials </li></ul><ul><li>Designed for agile evolution </li></ul>
    13. 13. <ul><li>Systemic approach and adaptability </li></ul><ul><li>Generic operational model (for business) </li></ul><ul><li>Advanced multi-layer model (for IT) </li></ul><ul><li>New features are added like pieces of Lego </li></ul><ul><li>Proven that BPM systems can be improved faster, better and less expensively </li></ul>Main characteristics of the architectural framework
    14. 14. Typical service and process oriented enterprise Business events : requests, payments, etc. Business processes Business events : offers, invoices, etc. Dept. Dept. Dept. Dept.
    15. 15. The simplified multi-layer model Execution Rules Objects Data repository repository repository repository
    16. 16. Why this approach produces agile systems <ul><li>Many of the difficult issues are resolved: </li></ul><ul><ul><li>Architecture, Methodology, Patterns </li></ul></ul><ul><li>There is no “classic” application – instead, there is a set of orchestrated services </li></ul><ul><li>Services those are versionable and clonable </li></ul><ul><li>The business logic is kept in one place </li></ul><ul><li>This approach has proven itself: production system in place for several years; several successful (easy to do) migrations undertaken </li></ul>
    17. 17. Works with other technologies Portal Grid infrastructure ESB infrastructure Business data services Business objects services Business rules services Business execution services Business measurement services Business intelligence services
    18. 18. Agile implementation of a new functionality <ul><li>A new functionality is generally implemented across systems </li></ul><ul><li>Any missing blocks are created in a dynamic language (e.g. Jython) and they are wrapped into services </li></ul><ul><li>It generally does not have its &quot;own&quot; database </li></ul><ul><li>It is &quot;outside&quot; existing systems </li></ul><ul><li>It reuses existing services </li></ul>
    19. 19. A common tool with BPMN and BPEL (e.g.
    20. 20. Many thanks to Jython <ul><li>Excellent as the glue between enterprise applications </li></ul><ul><li>Easy to manage (e.g. fragments are kept in .jar) </li></ul><ul><li>Highly flexible (i.e. introspection) </li></ul><ul><li>Dynamic loading and execution </li></ul><ul><li>The only thing that was added: </li></ul><ul><ul><li>a .py wrapper to simplify execution </li></ul></ul>
    21. 21. Real agility achieved: two types of project <ul><li>Micro-projects – agile implementations of new features </li></ul><ul><li>Meta-projects – architectural framework governance for the management of many micro-projects </li></ul><ul><ul><li>looks like maintenance rather than development </li></ul></ul>
    22. 22. Meta-projects are carried out in a manner similar to Deming’s wheel <ul><li>Plan </li></ul><ul><ul><li>fact- and rule-based selection of what should be done next as a micro-project </li></ul></ul><ul><li>Do </li></ul><ul><ul><li>execution of a micro-project </li></ul></ul><ul><li>Check </li></ul><ul><ul><li>new findings and solutions are considered for wider use </li></ul></ul><ul><li>Act </li></ul><ul><ul><li>refactoring of the system </li></ul></ul>
    23. 23. Management of micro-projects <ul><li>In-depth knowledge of the domain is essential </li></ul><ul><li>Sharing &quot;the vision&quot; with the business process owner </li></ul><ul><li>Architecting the product (i.e. a business process) in terms of IT and business systems </li></ul><ul><li>Guiding the project team to implement the product </li></ul><ul><li>Giving practical help, if necessary </li></ul><ul><li>Facilitate, influence and coordinate rather than control and act as the ultimate authority </li></ul>
    24. 24. Micro-projects: definition phase <ul><li>Business optimisations are evaluated </li></ul><ul><li>Features to be implemented are understood </li></ul><ul><li>Priorities (availability of features) are communicated </li></ul>
    25. 25. Micro-projects: specification / conception phases <ul><li>A prototype of a product (e.g. an executable business process diagram) is used to validate what will be implemented </li></ul><ul><li>Missing components (if any) are identified and specified </li></ul><ul><li>Missing components (if any) are evaluated; use of new tools/utilities should be justified </li></ul>
    26. 26. Micro-projects: development / test / validation phases <ul><li>Missing components (if any) are incrementally developed, tested and validated </li></ul><ul><li>Whole product is assembled from available and new (started as dummies) components </li></ul><ul><li>Whole product is incrementally tested, validated and deployed </li></ul><ul><li>Extra monitoring (if necessary) is deployed </li></ul><ul><li>Necessary resources are estimated and the system is reconfigured to provide them </li></ul>
    27. 27. Micro-projects: production phase <ul><li>New product or new version of a product is declared available for use by the business or by other components </li></ul><ul><li>In the case of replacement, some estimations are made when the previous version of a product is to be discontinued </li></ul>
    28. 28. Typical timing of micro-projects for standards production automation <ul><li>Definition phase: 1 hour </li></ul><ul><li>Specification / conception phases: a few hours </li></ul><ul><li>Development / test / validation phases: a few hours / days (depending on user’s availability) </li></ul><ul><li>Production phase: practically instant </li></ul>
    29. 29. Address some criticisms from the following book <ul><li>“ Extreme Programming Refactored: The case against XP” </li></ul>
    30. 30. “ Extreme culture” (jumping straight to code) <ul><li>There are no obstacles </li></ul><ul><ul><li>General design and patterns are available </li></ul></ul><ul><ul><li>Use a common tool for business process mapping </li></ul></ul><ul><ul><li>Can start with the existing process and refine it </li></ul></ul><ul><ul><li>A solution is firstly and quickly coded in executable business process diagramming language </li></ul></ul>
    31. 31. “ The on-site customer” (as a replacement of requirements) <ul><li>Close work with the customer is mandatory </li></ul><ul><ul><li>It is just about changing the IT attitude towards the users </li></ul></ul><ul><ul><li>From master to service provider </li></ul></ul><ul><ul><li>From talking 95 % about IT issues and 5 % about business issues to a 50 % - 50 % balance </li></ul></ul><ul><li>Follow one of the principles of the Toyota Production System (TPS): </li></ul><ul><ul><li>Go and see for yourself to understand thoroughly the situation </li></ul></ul>
    32. 32. “Pair programming” (for compensating absence of design, documentation, etc.) <ul><li>From permanent pair programming to systematic code reviewing </li></ul><ul><ul><li>Code review is necessary to keep conceptual integrity </li></ul></ul><ul><ul><li>Programming is a way for communication between humans </li></ul></ul><ul><ul><li>It is a good way to involve junior programmers </li></ul></ul><ul><li>Follow one of the principles of the TPS: </li></ul><ul><ul><li>Grow leaders who thoroughly understand the work, live the philosophy and teach it to others </li></ul></ul>
    33. 33. “Oral documentation” (documentation in XP tends to be paradoxical and confusing) <ul><li>A lot of documentation is already available, but often IT rejects it: </li></ul><ul><ul><li>Architectural design document is too complex </li></ul></ul><ul><ul><li>Workflow diagrams are too much business </li></ul></ul><ul><ul><li>Jython code requires some formal training </li></ul></ul><ul><ul><li>Quality documentation (considered bureaucracy) </li></ul></ul>
    34. 34. “ Constant refactoring after programming” (If it ain’t broken, fix it anyway) <ul><li>Any enterprise system is never finished </li></ul><ul><ul><li>Business and IT meaning of “broken” are rather different </li></ul></ul><ul><ul><li>Features are added if needed </li></ul></ul><ul><ul><li>It is a very dynamic system </li></ul></ul><ul><li>Follow one of the principles of the TPS: </li></ul><ul><ul><li>Become a learning organisation through relentless reflection and continual improvement </li></ul></ul>
    35. 35. Lessons learnt <ul><li>Combining the architectural framework and agile software development allows building of agile enterprise systems </li></ul><ul><li>The use of common tools by both the business side and the IT side is of great benefit </li></ul><ul><li>Neither agile development nor any architecture works well if the politics don’t fly </li></ul>
    36. 36. If the politics don’t fly, the system never will <ul><li>Politics, and not technology, sets the limits of what technology is allowed to achieve </li></ul><ul><li>Cost rules </li></ul><ul><li>A strong, coherent constituency is essential </li></ul><ul><li>Technical problems become political problems </li></ul><ul><li>The best engineering solutions are not necessarily the best political solutions </li></ul>
    37. 37. THANK YOU <ul><li>Questions and answers </li></ul>
    38. 38. Short description <ul><li>The aim of this talk is to share my experience in the use of a practical architectural framework for the improvement of enterprise business systems. This framework uses a dynamic language (i.e. Jython) and agile development practices. Experience with business acceptance and composition of applications are discussed. </li></ul><ul><li>This presentation is complementary to my presentation at Plone conference 2005 in Vienna (“The use of Plone for enterprise solutions”) – it is focused on the technical and practical aspects of agile evolution of enterprise </li></ul>
    39. 39. Architectural framework experience (1) <ul><li>Architectural framework provides a basis for technical and political decisions </li></ul><ul><li>IT development is unified </li></ul><ul><li>The target is business process automation and not applications development (brings greater flexibility) </li></ul><ul><li>The system is built mainly from commodity functional blocks (commercial and freely available) that are not modified </li></ul>
    40. 40. Architectural framework experience (2) <ul><li>The system is built incrementally (quicker ROI) </li></ul><ul><li>High level of user involvement </li></ul><ul><li>Team has ownership and customizes its approach </li></ul><ul><li>Project retrospection (no post-mortem or witch hunt) </li></ul><ul><li>Achieve synergy between practices, principles and experiences for building software systems </li></ul><ul><li>Take advantage of the organisational business and organisational knowledge </li></ul>
    41. 41. Based on SOA: In general, a layer, a building block and a version of a building block are all services WSDL GUI (optional) Service logic Service database (optional) Other services (optional)
    42. 42. Micro-projects: documentation <ul><li>Built-in quality management system </li></ul><ul><li>Use of visual tools (e.g. business process diagrams, forms) </li></ul><ul><li>User management, maintenance and programming </li></ul><ul><li>IT-specific documentation is mainly available from the meta-project and adopted patterns </li></ul><ul><li>Programs as documents and documents as programs </li></ul>