Descending from the Architect's Ivory Tower
Andrew Rendell
Principal Consultant
Valtech
andrew.rendell@valtech.co.uk
Abstract The requirement to design, document and review
This experience report, by a project’s technical everything upfront as a way to reduce risk is one that is
architect, details the adoption of Agile methods across obviously eschewed by the Agile movement as being
several teams after one high profile success. The ineffective and providing only an illusion of control. In
organization had a long history of waterfall addition, making a documentation quality gate the
development and a clearly defined remit for technical mechanism by which the technical architect manages
architects. Years of refinement had led to a set of the implementation paradoxically reduces their
techniques which contradicted many of the ideals held effectiveness by:
by Agile practitioners. The author’s challenge was to Isolating them from the technical implementation for
maintain agility and fulfill responsibilities inherited which they are supposedly responsible.
from waterfall processes without reverting to the Reducing their technical ability by taking them away
conventional practices that ultimately lead to the from the technology that made them great
architect’s ivory tower. candidates for the role in the first place.
Supporting the fallacy that the technical architect is
1. Introduction the all-knowing center of the technical universe.
Many technical architects on waterfall projects do
The T-Mobile International Mobile Portals and an excellent job. The author's opinion is that this is
Content Delivery Group had developed a set of achieved in spite of rather than because of the design
expectations around the responsibilities of a role which review quality gates. Personal experience is that many
would be widely recognized as a 'Technical Architect'. technical architects feel disenfranchised because their
The exact remit of a technical architect is subject to key skills honed through many years of education and
much debate and differs widely between organizations hard project work are no longer put to good use.
and even projects. This report does not aim to produce Conversely, many developers perceive the technical
a definition for the technical architect, Agile or architect as being somebody they rarely interact with,
otherwise, but in the context of this particular client who has little idea of how the software is being put
engagement the role involved having: a wide remit together. In many teams the author has observed the
over the implementations delivered by several different architect as being regarded as at best a poorly informed
teams; end to end technical responsibility; delivery of a individual, dabbling at the periphery and at worst, a
consistently efficient implementation that is fit for dangerous impediment to progress whose attention
purpose. must be avoided at all costs.
In organizations where a waterfall process is in Technical architect is usually a logical career
place it has been the author's experience that the progression for developers. The organization's need for
technical architect is unlikely to be involved in the an individual to fulfill the responsibilities of the
hands on aspects of delivering solutions. Their role is architect has not diminished, [1] even if the tools
often very closely coupled with design documentation. employed in the past have sometimes failed to add
Often the delivery process has been structured to value. The challenge is: how can the responsibilities of
include quality gates where the deliverable for the next a technical architect be fulfilled without introducing
stage is documented and then reviewed by the technical practices which reduce the agility, and therefore the
architect. The review of design documentation is the effectiveness, of the team? Valtech has demonstrated
primary tool available to the architect. This had been that by changing the techniques and attitudes of the
the case at T-Mobile. architect it is possible to meet this goal. This
experience report details practices employed by a

technical architect and his team across a body of work Code style was diverging. It was very easy to see
consisting of several projects. These practices are which team had produced any one piece of code as
evaluated in retrospect to measure their effectiveness. they were radically different.
Common implementation patterns that had been
2. Initial Agile adoption very well understood were not being applied
consistently. This led to a number of situations
In the summer of 2007 a marketing initiative for a where the application had previously been
new mobile portal was proposed. The adage 'necessity predictable now behaving in an unpredictable and
is the mother of all invention' applied to this project. inconsistent fashion.
The high profile and reduced time-scales (twelve Technical debt was increasing. A key development
weeks rather than six months from initiation to go-live) principle was that the code base should be under
meant that light-weight technologies and Agile constant rationalization to remove duplication and
practices had to be used rather than the incumbent redundancy and increase reuse and consistency. This
document-centric waterfall processes. This was very principle allowed the team to build fast, prove or
much a tactical Agile adoption. Failure was not an disprove a feature and then refactor to pay back the
option and the focus was on effective delivery rather technical debt. A technical backlog was maintained
than best practice. and tasks were regularly executed from this backlog.
The project was a high profile success in a very The net effect should have been a reduction in lines
short time scale. The delivery date was met, to the of source but analysis showed that as quickly as
hour. The team had proved that a number of Agile code was being refined, new code was being created.
techniques were highly beneficial and instilled Code was still being produced quickly but the debt
confidence throughout the group that more was not being repaid. The team was moving from a
comprehensive adoption was not only possible but proactive refactoring regime into constant fire
desirable. fighting mode with decreasing feature delivery.
During this period the architect was made less
3. The post launch doubts and scaling up effective by two critical factors:
The architect was on the critical path for code
After the initial euphoria of launch there were delivery on one of the projects with the same
doubts expressed by senior members of the expected ideal engineering hours capacity as any
management team. There were concerns that not all of developer.
the old practices should have been discarded. Of The architect's remit was well understood but the
particular concern was the lack of accessible mechanisms by which that remit would be achieved
documentation to allow maintenance of the platform, were not. This was the one of the traits of the
especially if the development team was cycled. This conventional ivory tower architect: responsibility
resulted in pressure to revert to some of the original with no clear mechanism of control.
document centric processes.
A victim of their own success, the team was now 4. Redefining the approach
required to deliver more features to the same high
levels of quality in the same aggressive time-scales. The technical architect, who had been aware of
Given increasing scope and fixed delivery dates the these issues for several months but had been unable to
only solution was to increase headcount. The team was correct them, not least because of his own development
grown and reorganized into two separate groups with commitments, determined that the situation required
different functional responsibilities in the same immediate and fundamental intervention. At this point
platform. The technical architect retained his position external events required the teams to be reorganized
across the team. into a number of different projects.
The two groups were seated in different locations. The technical architect took this opportunity to
Other than an hour long weekly team meeting there reorganize technical governance. The new regime
was increasingly little interaction between the two would allow scaling of development capacity through
groups. This led to the creation of silos where delegation and empowerment. To ensure quality and
developers in one group knew very little about what consistency the regime would gather and analyze
was happening in the other. empirical evidence. The architect determined that it
The new organization began to deliver quickly and was impossible to maintain the commitments of being
was generally viewed as a success but internally a full time developer and fulfill the technical architect's
several disturbing issues were surfacing: remit. New techniques minimized the architect's
isolation from the implementation, without the

unachievable requirement that the architect write key the project. If left they may well have spawned a
code modules. large number of similar features which would have
The following sections describe some of the main increased the technical debt.
techniques employed in this new technical governance The developers' confidence was boosted. They were
regime which kept the architect 'out of the ivory tower'. now sure that they were interpreting the
development guidelines correctly and had been
4.1. Reflecting on code structure publicly credited as such. The next retrospective
recorded that the developers regarded the code
During the initial, problematic, scaling up of the review as being one of the positive features of the
team, development patterns and priorities had been sprint. The architect going through code leaving
communicated. These were not always followed. In the TODO annotations etc. increased the sense of
new approach, after communicating the intent, the common code ownership. It made everybody more
realization was evaluated for compliance. This took the aware that the source was not an opaque artifact
form of detailed code reviews during the second whose functionality was the only facet that would be
iteration of the new projects as the body of code began observed.
to increase. One problem as the code base increased was
On a clean workstation, using only the instructions choosing which part of the application to inspect. One
on the wiki, the architect built a development technique that proved effective was simply to conduct
environment. This included configuring the Eclipse an exercise in the retrospective where each developer
IDE and Maven as well as checking out code and named their most complex or cleverest code module.
setting up development application server instances. These modules became candidates for detailed
This was an essential first test of the stability and inspection.
accessibility of the code base. This mechanism of code review did require a
The architect used a combination of Eclipse's significant investment in time. These reviews were
powerful code navigation tools and the acceptance and only conducted a handful of times over several months.
unit tests to traverse the application. After identifying The prohibitive cost of these detailed code reviews
the classes involved in particular user goals, a UML meant that more commonly developers were invited to
tool reverse engineered the code into a set of class use a whiteboard to talk the architect and a number of
diagrams. The architect used Eclipse and the UML tool their peers through the interactions and class structure
to determine the associations between the of a particular section of the code. The objective was
implementation classes and their tests. The architect much the same as the detailed code inspection but also
then examined the implementation of the unit tests and served to educate a wider audience. Whilst it was of
made brief passes of the code to determine the comparable in cost to the project in man hours the cost
responsibilities of each class. This exercise indicated to any one individual, critically the architect, was
whether standard patterns and agreed libraries were in reduced. E.g. a white-board session would require two
use. Importantly it articulated the class cohesion and hours preparation by a developer and then one only
structure. hour for attendance from the architect, two other
Issues were identified around encapsulation and developers and the developer presenting. The cost is
cohesion. Anti-patterns in test classes which indicated six hours to the project but only one hour is taken from
issues in the implementation were noted and verified. the architects diary. A more effective but costlier code
The exercise produced a list of issues to be inspection might easily cost six hours of the architect's
corrected. The architect annotated the code in several time.
places with FIXME and TODO and finally produced a White-board sessions were less useful than code
UML class diagram with notes showing the class inspection. They did not bring the architect and other
structure in use. This was added to the wiki. developers into close contact with the actual code. It
The exercise formed the basis for several often led to a level of abstraction (consciously or not)
improvement points at the next retrospective and being introduced by the presenter in order to
allowed the architect to provide positive feedback on communicate with the audience. Occasionally it
the implementation based on real evidence. appeared to reduce the sense of common code
This exercise had several positive outcomes: ownership as one individual became the recognized
The architect's confidence that the team was expert.
following the correct and consistent set of patterns
was firmly established. The architect also gained
valuable familiarity with the code base. The issues
that were spotted were easy to correct at this stage of

4.2. Providing direction and rigorous to the network team, the latter to the owner of the
checkpoints integrated system). This was stipulated by the
Development Principles but had been missed during
The architect created a Development Principles development. The audit picked up these sorts of issues
wiki page which was then presented to the team in an which previously had only been uncovered in UAT or
interactive session. The principles were deliberately not production.
generic points that could be universally applied on any The cost of the audit was high. It was expensive to
project. Instead, these principles were derived from the develop and maintain and required significant input
best working practices used on the T-Mobile portal from project members whose time was heavily in
application. They were very specific and easy to apply. demand. Scheduling was difficult and audits were often
The principles covered a wide range of topics from delayed, which increased risk to the project. The audits
TDD and patterns for concurrency to policies regarding were enormously beneficial and fully justified their
the team’s attitude to broken builds. They also high cost. They provided valuable empirical evidence
contained sections that related to common functional and reduced the architect's isolation from the
requirements such as error handling. implementation.
Reviews and retrospectives supported the view
that these principles had a positive effect encouraging a 4.3. Structuring the application architecture to
consistent approach. promote good governance
In keeping with the approach of gathering
empirical evidence over reliance on documentation, the It was found that restructuring the application
Development Principles were supported by an audit architecture was a contentious but effective tool to
driven by an Architecture Checklist. Like the improve technical governance.
Developer Principles the Architecture Checklist was The application was designed from inception with
developed specifically for this suite of applications. a clear modular structure with loose coupling. Events
The temptation to try and make a generic tool which had demonstrated that it was still possible to build
could be widely reused was resisted. components that violated encapsulation by the
The audit of the system using the checklist was corruption of shared services or simply by consuming
performed by the architect and technical lead paired at all the CPU or memory allocated.
a workstation. The code was checked out clean and the When the new projects were initiated the
IDE and test framework were used to check various assumption was that they would all be extending the
points. In previous projects the author had experienced existing application. The architect determined that this
audit exercises driven by a review of design made the technical governance more difficult. Towards
documentation followed by an interview. This required the end of the first iteration he proposed a departure
overly time consuming preparation, was stressful and from this architecture. The monolith would be replaced
less effective than inspecting the code and running several discrete applications. Where the same code was
tests. required in more than one platform this was moved into
Since the check list was developed for this shared libraries (distributed and controlled via Maven)
application suite most points were pertinent. Each which individual applications could branch if required.
check was phrased as a question where a given This move had a significant positive effect. The
response would sometimes indicate that a more teams were decoupled in the same way as their
detailed section was applicable. Not every question applications. The silos that had been in effect
was answerable by a simple yes or no, instead where previously were recreated but with clean interfaces
appropriate the reviewers recorded a written answer. which could be easily policed. Developers now had
This formed part of the documentation and most freedom to innovate rather than a license to interfere
importantly stimulated deeper inspection. and disrupt.
The audit exposed the architect and technical lead Although their deployment workload had been
(who may or may not have been a senior developer increased, the operations team was supportive because
depending on the project) to the code. It was not a they were given better performance testing guarantees.
foolproof tool and obviously did not detect all errors. It One of the failures late in 2008 had been caused by a
did expose issues that would have been show-stoppers presentation layer module consuming unacceptable
later in the application lifecycle. E.g. an integration levels of CPU capacity. Since all modules ran in the
module was found to report a network connectivity same container it took over a week of repeated tests to
error to the operators in exactly the same way as an ascertain that the complex integration modules,
unexpected response from an external system. These obvious candidates for extreme CPU use, were not at
errors required different escalation routes (the former fault. The updated architecture allowed each

component to be load tested in isolation. This meant reduce the distance between the architect and the
that issues were identified with fewer test cycles. implementation [1].
Simplification and encapsulation of the Definition of load testing profiles (i.e. agreeing
implementation directly led to more effective what constituted 100% load), detailed review and
architectural governance without imposing onerous coordination of load test execution were key
processes. Although the initial emotional response to responsibilities of the architect. The cost of these
this change was that it would be very costly, in activities was extremely high but entirely justified by
retrospect, even though it was initiated in the second the direct exposure it gave the architect to the non-
iteration it still only required an additional ten days of functional aspects of the system. These aspects are
development time. It saved many times that effort by fundamental in delivering the architectural remit. It
removing the requirement to regression test alone. was found that the architect and technical lead would
be forced to concentrate solely on load testing for long
4.4. Key measurement tool: testing periods. This came at the cost of ignoring the demands
of the other projects during these times. It was a
The technical architect had always placed a high significant issue if these load tests occurred at the same
value on a test first strategy and the adoption of TDD at time as other critical activities (such as retrospectives
T-Mobile was the subject of an Agile 2008 Experience or sprint planning) for other projects.
Report [3]. All projects had a reasonable unit test JMeter and use case test execution gave the
coverage (60% lowest to 85% highest). Unit tests were technical architect a high level of confidence that the
written by and were mostly for the benefit of the applications were fit for purpose based on empirical
developers. evidence and real experience rather than
A second class of tests, labeled acceptance tests, documentation.
were closely aligned with the user goals of the system
and were intended to be developed in conjunction with 4.5. Documentation that is 'good enough'
the proxies for the business stakeholders
(organizational issues precluded the direct involvement In keeping with Agile values, the technical
of the stakeholders themselves). architect was determined not to expend valuable effort
On some projects resource constraints meant that producing documentation with no clear purpose when
the acceptance tests were often created by the that effort could be better used to bring delivery closer.
developers without the involvement of other At the same time the architect wanted to ensure that
participants. Whilst these tests still had significant where documentation was genuinely required it was fit
value, an opportunity for the architect and proxy for purpose.
stakeholders to verify that the system was fit for One project delivered a set of web services. A
purpose as they understood it was missed. As the tests comprehensive, example based, API specification was
were developed solely by engineers the technical produced. This document was as formal as any
complexity of the code inhibited comprehension by document delivered by waterfall projects at the client.
proxy stakeholders. This document was identified as being critical to
To mitigate against the above issues the architect success and therefore justified its high cost in man
initiated the development of a new class of tests, hours to write and review.
referred to as use case tests, which ran against the Previously a design document for each module had
application fully deployed. These tests were supported been mandated. This rule was discarded. Instead, key
by a simple framework which aimed to make the tests areas were identified by the architect or the team as
themselves resemble the language of the interface being important or complex enough to justify some
specifications, i.e. the tests were expressed in a form of design review and capture. White board
language that the stakeholder proxies could understand. sessions were led by the appropriate developer. These
These tests became a powerful tool for measuring were captured using digital cameras and uploaded to
completeness against functional requirements. The the wiki along with a brief summary of any conclusions
construction of this test suite highlighted several areas or activities to complete. Where an area was identified
where the implementation had diverged from the as particularly important the architect had formal UML
published API. These tests also provided a seed into diagram production added to the sprint backlog. This
the creation of load tests (using JMeter). ensured key documentation was completed, its cost
The technical architect helped construct and run was visible and that cost was not absorbed into the
these tests rather than relying on a report from others. development activity. These documentation tasks had
This practice was prioritized as another mechanism to specific goals, e.g. communicate the lifecycle of an
object through a state diagram such that the design can

be verified against use cases. The UML diagrams were allow developers to take the lead in producing the
held in a single, source controlled, highly accessible solution with minimal guidance. This allowed best
UML repository. practices to be developed by individuals and then
Given the client's history of a document centric adopted across the teams.
process the new approach to documentation was Ivory tower architects often concentrate on
always going to be contentious. This was especially technology rather than people. This stops them
true when the development process had some sort of delegating and therefore impedes the development
interaction with the wider organization. A security scalability.
audit was performed several months into one project's
life by an external group who had been informed that 5. Conclusions
they were dealing with an Agile team. Due to some
inter-programme communication issues, they were An architect must reduce their isolation from the
only supplied with a couple of powerpoint slides. This implementation by being closely involved with high
met all their preconceptions of Agile. The auditors value technical activities such as load testing and
were surprised when the architect was able to supply code reviews.
on demand a number of succinct and appropriate An architect can become isolated from parts of the
documents. These were generated from the UML system if they cannot find time to cover all areas
repository or copied from the wiki but were exported because they are attempting to also be a full time
into a company standard document repository to developer.
comply with versioning and accessibility rules. This Relying solely on documentation as a tool for
demonstrated to the security auditors that the project technical governance is not an effective strategy but
was as rigorous as any of its waterfall peers. there are documents (which may be on the wiki or in
the UML repository) which are essential to the
4.6. Delegation architect.
Learning to employ ‘soft’ skills becomes as
There is always going to be a point where it is important as technical acumen because without
impossible to achieve any more development excellent communication and effective delegation
throughput without adding more people to the you cannot scale up.
equation. It is the familiar pattern of horizontal Automated, well written tests are the architect’s best
scalability always outperforming vertical scalability at mechanism to gather empirical evidence of
some point. compliance with governance and fitness for purpose.
Agile projects empower developers. The best techniques cannot be applied every time
Empowerment requires delegation. To be able to because of cost. Choose the most important areas for
delegate tasks you need to have a team which is fit for the high cost activities and use less effective but
purpose [4]. As part of the client's rigorous selection cheaper methods for others.
procedure the architect performed a technical Activities that made the developers aware that the
assessment of all new joiners with a development architect was observing the source increased the
remit. In the course of his career the architect had perception of common code ownership and
observed many interviews being concluded using invigorated developers to maintain high levels of
emotional rather than empirical methods. code quality.
The technical architect developed a set of case
study driven interviews customized for the T-Mobile 6. References
project. This meant that the candidate could be
exercised using the project's working practices and [1]V. Hazrati, The Shiny New Agile Architect , 2008
technologies. Interviewees were expected to run white [2]J. McGovern, A Practical Guide to Enterprise
board design sessions based on common problems the Architecture, 2003
project faced or use TDD using Maven and Eclipse.
[3]A. Rendell, Pragmatic and effective Test Driven
The interviews were designed to give candidates a
Development, 2008
vehicle to demonstrate their abilities rather than trying
to trip them up. [4]S McConnell, Rapid Development, 1996
It was the architect's opinion that this was a critical
factor in assembling a strong team whose practical
ability had been proven before they started the project.
This enabled a high degree of immediate delegation
and empowerment. The technical architect was keen to