Tech challenges in a large scale agile project

Tech challenges In a Large-Scale Agile Project Harald Søvik and Morten Forfang Computas AS, Lysaker Torg 45, 1327 Lysaker, Norway {hso,mfo}@computas.com, http://www.computas.com Abstract. A five year, 25 man java project effort, that started with a waterfall-like methodology and that adopted Scrum after less than a year, has been concluded. We present three key technical challenges, briefly analyze their consequences and discuss the solutions we adopted. Firstly, we discuss how we modularized our architecture, module delineation principles, coupling and the trade-offs of abstraction. Then we discuss testing environments, their automation and relation to branches and the effect on customer involvement and feedback. Finally we discuss the benefits and disadvantages of representing domain knowledge declaratively. For all three challenges we discuss how the project’s agility was affected. Key words: java, scrum, agile, modularization, coupling, technical challenge, module bloat, dependency, applicaiton layer, pojo, ide, testing, branch, continuous integration, deploy, maven, database updates, domain expert, footprint, declarative knowledge, business logic, domain knowledge, process 1 Introduction A 5 year long Java and agile-project has recently come to a conclusion. It has been conducted between Computas (a Norwegian consultancy firm) and a major Norwegian governmental authority. As a rather large project (manning 20-30 developers throughout the project), it has faced a lot of challenges, but overall the project is recognized as a major success. This experience reports focuses on three major technical challenges that arose during the project. We believe those challenges partially can be attributed to the SCRUM methodology. For each of them, we try to identify the consequence and the cause, and then follow up with any solutions we tried, and an analysis of whether the problem was successfully solved or not. 1.1 A brief history The project started out using a crossover variant of the waterfall methodology and long-term iterations, but turned towards agile after 8 months of development. The customer and domain experts needed to see, touch and feel the system

2 Harald Søvik and Morten Forfang being developed. A lot of them were previously not familiar with software development. Being able to see results of their decisions quickly made them more confident and eager. To support parallel development, the project was organized into 4 different teams, whereas 3 were committing to the development branch, and one team working maintenance on the production branch. The teams were ”self sufficient” in technical expertise - meaning that multiple programmers could be working simultaneously on the same module or layer of the system. The teams were also assigned a pool of domain experts suitable for the tasks they had chosen for the next iteration. The project were divided into 4 overall domain-centric periods, each support- ing a specific business target. All of these focused on similar goals: – Enable self-service or more automated customer interaction – Reduce the number of internal databases and managed systems – Increase data quality – Unify business processes to ensure quality 2 Coupling and modularization The dependency topology of a system and it’s relation to evolvability is a cur- rent, active research field ([3]). It is predominantly argued that modularization, low coupling and high cohesion are characteristics that enhance salient non- functional characteristics of a system, like analyzability, integrity, changeability and testability ([4], [7]). In this section, we explore some of the trade-offs we have found between modularization and various forms of efficiency. We use the terms subsystem, cohesion, coupling and framework as defined by [6] and component and collaboration as defined by [8]. 2.1 Subsystems and frameworks The project consists of a series of core subsystems and legacy systems. In this article we focus on the Mats subsystem. This subsystem depends crucially on the Framesolutions framework. The Framesolutions framework is a fairly comprehensive set of customizable, proprietary components that many of our projects use. It at the same time provides core functionality on all application layers and it provides a fairly rigid set of programming patterns. The framework is centrally maintained separately from the project source code. 2.2 The Mats subsystem For the purposes of this article, the The Mats subsystem features four build-time modules; the application server, the web server, the client and a common module. The run-time and compile-time dependencies between these are illustrated in figure˜1.

Tech challenges In a Large-Scale Agile Project 3 Fig. 1. The build time and runtime dependencies between important Mats subsystem modules. Technologically, the two first are realized as Java enterprise archives running on JBoss 4.2.3 servers, the client is a Java application and the common module is a jar file included in all the other modules. Overall, this particular way of modularizing the Mats subsystem strikes a good balance between having increased complexity through many modules, de- creased intra-module coupling and increased module cohesion. There are however a couple of major ill effects: – The common module got bloated. – New, distinct functional areas were not handled well enough as separate collaborations with proper dependency management. Let’s first briefly discuss the module bloat. On the surface of it, there are few reasons why one shouldn’t put a piece of code or resource in the common module. It is always present and the developer doesn’t have to think about possible reuse. As long as the entity in question is put in common, it can always be reused by some other module. This idea were thought to be a very good one in a large project, and were even more so thought of when we turned agile. We believed that an large and agile project should have a lot of supportive means to enhance reusability. When multiple teams were working on multiple tasks, it could be difficult to spot a candidate for reusability. But if the code was put in the very same module, we believed the programmers could hardly overlook the ”reusabilityness”. The idea were not very successful, and while reuse inside a specific sub-domain (read: team) sometimes has been possible, cross-domain (read: cross-team) reuse is virtually non-existent. This leads to a bloated common module, lowers cohesion and heightens coupling. In runtime we increase the overall footprint since the common module is present with all the other modules. At build-time we increase the build time

4 Harald Søvik and Morten Forfang since the common module is bloated. Finally, each of the non-common modules become unnecessarily complex since there is much in the common module that is unwanted and unneeded by the module in question. Now, let’s move onto dependency management for new collaborations. Given some functionality you may choose to put this into one of the four mentioned modules, but you cannot choose to put it in a module governed by functional area. For example, if you are writing some web functionality from the farm animal domain, you can choose to put the code in the web or the common module, but there is no module that takes care of the farm animal domain. This leads to modules being characterized by having general purpose and very domain speciﬁc code side-by-side. Clearly this makes the modules lower their cohesion and calls for sustained developer discipline to avoid increasing intra- module coupling. With the comprehensive, automatic test approach and environment in the Mats project, we categorize the tests according to test level and domain. Char- acterizing tests by domain alleviates the problem of discerning which functional area a particular piece of code belongs to. This doesn’t improve module coupling, but it makes maintenance and refactoring easier. For example: @Test(groups = {"unit", "husdyr"}, dependsOnMethods = {"validererNyIndivid"}, expectedExceptions = IllegalArgumentException.class) public void kanIkkeMerkesSammeDagenEllerTilbakeITid() {} (from Mats 4.0b-SNAPSHOT depending on testng version 5.8) Here “husdyr” (farm animal) is a category corresponding to a fairly large functional subdomain. 2.3 Application Layer Separation From a developer’s technical perspective we let the Mats subsystem (cf. section 2.2) be a single project. The Framesolutions framework and code pertaining to the Integration Platform is kept apart. Modern IDE’s are getting quite good at providing instant and global symbol search and dependency tracking1 . This means that all the previously mentioned modules appear as one seamless whole. The FrameSolution framework (cf. section 2.1) provides two quite powerful mechanisms that makes the Mats subsystem (cf. section 2.2) module boundaries easier to handle. There is the concept of a FrameSolution Manager. Such a manager serves as a “bridge” over a layer or between modules. The bridge may well cross physical hosts and application servers. An example invocation from Mats is ImportMeldingManager.getInstance(). getSisteFoerstemottakerAdresser(importoer, foersteMottaker) 1 The Mats project is predominantly using IDEA IntelliJ version 8

Tech challenges In a Large-Scale Agile Project 5 This code will look identical in all four Mats subsystem modules. For example, if this code is run on the web server, it will remote call a server stub on the application server. This code is easy for a developer to understand, quite condensed and not far from the ideal of a POJO method invocation. There is just this line needed, no annotation or other configuration. There is the concept of a soft pointer that transparently deals with persistent objects that are not “owned”, i.e. that the association between a class A and a class B is not such that CRUD operations on one is continued on the other. The soft pointer mechanism is together with a cache, implemented such that no matter where in a virtual machine an instance is soft referred to, it always points to the same, single persisted instance. This removes the burden of keeping track of multiple editions of the same object. It furthermore makes it possible to make intelligent, automated decisions w.r.t. persistent cascading and lazy loading. Generally this particular configuration makes it very easy to create functionality covering the entire vertical stack from the GUI at the thick client or on the web through business logic and down to the persistence or integration service layer. The code looks surprisingly similar on all application layers. Developers are overall very happy with this arrangement. One would perhaps think that this great freedom results in the inter-layer APIs to be in too great flux and that the entire subsystem architecture would over time loose necessary integrity. Due to the rigour of the FrameSolution framework (cf. section 2.1) that enforces quite strong patterns on all layers, this does in general not happen. There are however still some major unintended effects: – Just looking at the code in the IDE, it is not always clear on what layer (client, web server, application server) one is coding. Due to the powerful Framesolution abstraction features, the code would work, but not surprisingly, it often leads to highly resource demanding code. – Since the entire Mats subsystem source is editable and available in the IDE, the build time on the individual developer host becomes intolerable. The IDE’s automatic synchronization features also take an irritating amount of time. These unintended effects were partially handled by using a hierarchical module structure, dividing all code into one of the Mats subsystem modules, cf. section 2.2. However, this was not a very effective layering, since a lot of code still resided in the common module. Other mitigating actions were to introduce faster developer hardware (par- ticularly SSDs) and optimizing the build procedure. It remains a problem, and a fundamental trade-off, whether to let the developer use the powerful abstraction features of the Framesolution framework or write more module dependent, efficient, lower level code. 2.4 Summary Overall we’ve found that developers are quite happy with the productivity boost they get from layer abstraction, having all the source in the IDE and the lack

6 Harald Søvik and Morten Forfang of bureaucracy when dealing with a small number of ever-present modules. We clearly could’ve introduced more modules, stronger barriers between them or different delineation principles. This could’ve bought us higher and better cohesion, and lower intra-module coupling. Chances are, we also would’ve introduced more overhead, increased inter-module coupling and increased demands on developer discipline. 3 Testing environments To support the idea of agile development with domain experts, several testing environments were accessible for the experts throughout the project. This was introduced as early as possible so that the experts could get comfortable with the system and the quirks of using a system that is under development. We were working with two branches of the code (most of the time), and both of these had a range of environments associated with them. First, the development (”trunk”) branch was the target of all changes bound for the next major release. Second, the maintenance branch was holding error fixes and small changes that could be deployed to production very often. Each of these branches had three classes of testing environments: Continuous build [2], nightly build and manual, full-scale build. Thus, there were six different environments to maintain at all times. We did invest heavily in infrastructure and build automation for these environments. 3.1 Effect Because of the wide variety of testing environments, the experts always had an appropriate environment to carry out their testing needs. At first, it proved a little difficult to realize how to employ this diversity, and it was frequently necessary to remind them of in which environment it would be advisable to test a new feature or a bug fix. After the first major release, most people got acquainted with the idea, and became very happy with the different environments. See Table 1 for the different applications. Continuous Nightly Manual Development Experimental features New, stable features Production ready features Maintenance Experimental bug fixes Stable bug fixes Regression tests Table 1. Different testing environments A key feature in maintaining all of these environments, was to have com- pletely automated build- and deploy processes. It proved incredibly valuable - both for scheduled (nightly) builds, and whenever a new feature had to be tested immediately (continuous). To be able to automate the deploy process, it was close

Tech challenges In a Large-Scale Agile Project 7 to necessary gather all build steps under a single command. This was successfully implemented with Maven. This helped assure similar builds on multiple platforms, i.e. different integration environments and development platforms. 3.2 Difficulties Database structured updates proved to be difficult to automate, and were thus handled manually. It would have been possible to automate within the application, but the customer technicians discouraged such an approach. It was believed that an automated handling would weaken the exhibited control over database schema. After spending a lot of time reimbursing the necessity of continuous testing, many of the domain experts got used to the idea of having a environment at hand all day and night. This notion was also backed by a stable production system. Thus, whenever a testing environment failed to deploy correctly at night, or a programming error lead to a critical failure of the application - the domain experts would quickly let us know that they were inhibited from doing their work efficiently. To counter this, the service monitoring tool used in production were also employed to testing environments - with email and even SMS alerts. The testing environments were subject to the same uptime requirements as production. Of course - there were no really dramatic consequences when a fault lead to unplanned downtime. 3.3 Conclusion The biggest con of this approach was the impact on the development platform. Whereas nightly deployment worked fine, frequent deployments by developers or the continuous integration engine proved very time consuming. It is thus our conclusion that a more flexible and dynamic approach should be available for developers. This was partially implemented with hot code reloading tools, like Java hotspot and JRebel. These approaches were partially successful: Many changes were handled and deployed seamlessly, but changes to initialization code and structurally modeled data had to be handled otherwise. Testing environments and infrastructure is at heart of an agile project. The ability to quickly recognize new or altered features, and give feedback to appropriate developers or domain experts, is a stepping stone in enabling agility. 4 Declaratively represented knowledge A key concept in this project was Business Process Modeling: Separation between domain knowledge and implementation. From artificial intelligence, cognitive architecture work and knowledge modeling, the distinction between declarative and procedural knowledge has long been deemed important [1], [5]. This idea could be paraphrased as separating what to do from how to do it. It allows the

8 Harald Søvik and Morten Forfang domain programmers to work decoupled from the technical programmers. The processes can be modeled and mocked and remodeled, until the ideal workflow is reached. A few vendors offers systems implemented this way, and Computas is perhaps the leading supplier in Norway. We believed that this technique would fit quite well in an agile project. It would be possible to change the behaviour of the system without altering the technical details, and visa versa: Altering the technical implementation without altering the business processes. 4.1 Effect This approach presents challenges to the way a customer (and developers) man- age software. Since the knowledge is decoupled from the program logic, it is possible to alter the behaviour of a running program. Thus, whenever the developers decided to refactor the business logic, or the domain experts decided to alter the business processes, it would introduce no overhead for the other party. And this proved to be a correct assumption. Even though most of the business logic stayed the same once written, the business process often changed in subsequent iterations. This made the domain experts rater self sufficient in many questions, and made it possible to adapt to experiences in a quick fashion. However, we underestimated the need of technical means to handle this distinction: It should not be necessary to rebuild and redeploy such a system when developing and testing processes. Nevertheless, developers often did so to ”ensure correctness and consistency”. 4.2 Difficulties It might sound trivial, but postponing to figure out how to correctly reload process definitions at runtime reduced the development turnaround significantly. And not to mention the developers dissatisfaction of having to wait for changes to deploy before testing. Thus, it became clear that the application had to be de- signed for ”hot knowledge reloading”. This challenge was solved by implementing a re-initialization feature: Whenever process definitions or other domain defined data did change, the developer was able to reload the definitions and start using them right away. This feature is not used in production, but would theoreti- cally allow process managers to alter a process in the middle of the day without scheduling downtime and employing technical personnel. Also, our proprietary domain modeling language (although BPEL-compliant) were of course unsupported in any Integrated Development Environment (IDE). The language itself has a dedicated IDE which is well implemented and very functional for modeling purposes. But the functionality provided by our code IDE[9] were oblivious to such data representation, and provided no development support whatsoever. It hindsight, we should have realized the cost, and implemented a support plugin for our code IDE, so that it would treat changed to

Tech challenges In a Large-Scale Agile Project 9 process code the same was it treated technical code. That is: provide support for syntax highlighting, autocompletion and code deployment 4.3 Conclusion Use of proprietary data formats can both be rewarding and expensive. One need to be aware of the necessary level of support, and accept the cost of maintaining such a level. Use of declarative knowledge representations is potentially a major catalysator when dealing with domain experts, but actually implementing the knowledge requires a high level of expertise. Also in this matter, one should be very careful with respect to the tools one choose to use for implementation and assist. 5 Summary This experience report outlines three difficulties we experienced during this project. Although the details may appear minor, the impact of improvements was significant, because of the project size. We believe that it is incredibly important not to neglect those problems that were manageable in smaller projects, but may scale fast in large projects, and give the developers an arena where they can give an early warning of such challenges. We would like to emphasize some key issues from this report: – Abstraction and generalization should been driven by necessity and experience, not guesswork and intentions. – Abstractions is not a substitute for decoupling and formally controlled module interfaces. – Treat your testing environment like a small production environment. Define a Service Level Agreement for your domain experts and testers. Automate everything. – Do not underestimate the accumulated overhead caused by small problems being ignored over a lengthy period of time. – Do not underestimate the support that can be provided by modern day development tools. References 1. Brachman R. and Levesque H: Knowledge Representation and Reasoning, 1st ed., Morgan Kaufmann 2004, ISBN 978-1558609327 2. Fowler M.: Continuous Integration, http://www.martinfowler.com/articles/ contin- uousIntegration.html 3. Breivold H.: SOFTWARE ARCHITECTURE EVOLUTION AND SOFTWARE EVOLVABILITY, Malardalen University, Sweden, 2009

10 Harald Søvik and Morten Forfang 4. Breivold H. et al.: ANALYZING SOFTWARE EVOLVABILITY, 32nd IEEE Inter- national Computer Software and Applications Conference (COMPSAC), Finland 2008 5. Kendal S. and Creen M.: An Introduction to Knowledge Engineering, 1st edition, Springer 2006, ISBN 978-1846284755 6. Lethbridge T and Laganiere R., Object-Oriented Software Engineering, 2nd ed., McGraw-Hill 2005, ISBN 978-0073220345 7. Pressman R: Software Engineering: A Practitioner’s Approach, 5th ed., McGraw- Hill, ISBN 978-0073655789 8. Uniﬁed Modeling Language, version 2.x, www.uml.org 9. http://www.jetbrains.com/idea/

Tech challenges in a large scale agile project

by Harald Soevik on Jun 16, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Tech challenges in a large scale agile project — Document Transcript