Speeding up the production of software development is a typical request to software development teams. A naive solution to this problem is to simply double the people working in a team to consequently double the output of the team. However, as experience shows, this typically introduces problems like organizational overhead, more communication effort, poor common understanding of coding style and functional requirements, less people with a good overview, poor ramp up and knowledge transfer capabilities. Therefore, we need different, more feasible approaches. In this paper, we describe such approaches, more specifically, we describe how visualizations, communication tools, team size and team staffing influence software development speed. This paper provides hands-on experience on how to reach the goal of delivering working and well-crafted software in time and in a distributed working environment at Infonova. We identified the following key areas for successful scaling and distributing of agile engineering: (1) team organization, (2) infrastructure for distributed development and (3) automatic visualizations and feedback. Slides for a talk held in 2012 at the OTS Software Conference in Maribor (http://ots.si/Prejsnje_konference/OTS_2012/Povzetki.html#G_Oettl)

1. Lessons learned in distributed
agile engineering on a large
scale integration solution
DI Georg Öttl, Dr. Boštjan Grašič, Thomas Schenkeli, Dr. Stefan Klein

2. © 2011 INFONOVA GmbH 2
Infonova
Established in 1989
350+ Employees in Austria
300+ Employees in Graz
170+ Employees in „ISE“

3. © 2011 INFONOVA GmbH 3
Infonova’s R6 delivers Telco 1.0 and
Cross-Industry 2.0 Business Model Capabilities
Infonova R6
Business Architecture
Infonova R6
Process & Module Framework
Wholesale aggregation for multiple suppliers & Multi-tenant
Concept-to-Cash for Multiple Brands & Channel Partners
Secondary
Operating
Entity
Primary
Operating Entity
Secondary
Operating
Entity
Gateway
Water, Gas
Electricity
Fixed/Mob
Netco(s)
Cloud
Services
Media
Content
etc.
External
Retail BSS
Multiple Agents
A1 A2 An
Infonova R6
VSO1 VSO N
Product
Mgmt.
Customer
Mgmt.
Order
Mgmt.
Billing Finance
Platform
& Business
Mgmt.
Product &
Service
Management
Customer Self
Management
Ticket
Management
Customer
Relationship
Management
Service and
Resource
Provisioning
Order
Management
Rating
Invoicing
Bill Formatting
Mediation
Accounts
Receivable
Channel
Management
Supplier
Management
Virtualization and White Labeling
Integration Platform (Workflow Management, Task Management, Notification Management)
VSO1 VSO2 VSO N
Product
Mgmt.
Customer
Mgmt.
Order
Mgmt.
Billing Finance
Platform
& Business
Mgmt.
Product &
Service
Management
Customer Self
Management
Ticket
Management
Customer
Relationship
Management
Service and
Resource
Provisioning
Order
Management
Rating
Invoicing
Bill Formatting
Mediation
Accounts
Receivable
Channel
Management
Supplier
Management
Virtualization and White Labeling
Integration Platform (Workflow Management, Task Management, Notification Management)
VSO1 VSO2 VSO N
Product
Mgmt.
Customer
Mgmt.
Order
Mgmt.
Billing Finance
Platform
& Business
Mgmt.
Product &
Service
Management
Customer Self
Management
Ticket
Management
Customer
Relationship
Management
Service and
Resource
Provisioning
Order
Management
Rating
Invoicing
Bill Formatting
Mediation
Accounts
Receivable
Channel
Management
Supplier
Management
Product
Mgmt.
Customer
Mgmt.
Order
Mgmt.
Billing Finance
Platform
& Business
Mgmt.
Product &
Service
Management
Customer Self
Management
Ticket
Management
Customer
Relationship
Management
Service and
Resource
Provisioning
Order
Management
Rating
Invoicing
Bill Formatting
Mediation
Accounts
Receivable
Channel
Management
Supplier
Management
Product
Mgmt.
Customer
Mgmt.
Order
Mgmt.
Billing Finance
Platform
& Business
Mgmt.
Product &
Service
Management
Customer Self
Management
Ticket
Management
Customer
Relationship
Management
Service and
Resource
Provisioning
Order
Management
Rating
Invoicing
Bill Formatting
Mediation
Accounts
Receivable
Channel
Management
Supplier
Management

4. © 2011 INFONOVA GmbH 4
ALBTELECOM
Key customers

5. © 2011 INFONOVA GmbH 5
Scrum @ Infonova
 Scrum is iterative and incremental agile development method
 Three main roles: product owner, development team, scrum master
 Practicing Scrum since Oktober 2007
 ~170 Software-Engineers in 28 Teams
 Publications in technical journals and on technical conferences
 Scrum-Coaching
 Guest lecturers on Universities

6. © 2011 INFONOVA GmbH 6
Scrum @ Infonova
Sprint
Retrospective
Sprint
Planning Sprint Review
(Demo)

7. © 2011 INFONOVA GmbH 7

8. © 2011 INFONOVA GmbH 8
Key challenges
 How to manage a project that is being realized across different sites?
 How to achieve good collaboration and communication between
distributed teams?
 What skills do the team members need to be able to work in a distributed
environment?
 How can we build up new skills and share knowledge efficiently?
 What is the optimal team structure? Should the distributed teams be
isolated, partially distributed or fully distributed?
 What is the optimal team size for a cross-site team?

10. Practical Part
DI Georg Öttl
Senior System Engineer @ Infonova

11. © 2011 INFONOVA GmbH 11
 Distributed teams at Infonova
 Lesson 1: Team setup & team rampup
 Lesson 2: Visualization and automated feedback
 Lesson 3: Common understanding of code and code quality
Outline practical part

12. © 2011 INFONOVA GmbH 12
Infonova Teams
• Meta Scrum  Staffing
• Verteilte Teams vs isolierte Teams (Zugriff auf Jira von außen) /
Slides Thomas Schenkeli?
• Org Chart / Slide Matrix-Überlagerung von Stefan B.
• ISE Ziele (Craftsmanship / CI?) / Job Levels?

13. © 2011 INFONOVA GmbH 13
Development Sites Distributed Project
Team Setup
 Team started with 4
developers, one scrum
master-developer
 25 Team Members, 3
scrum master
 4 Sites
 2 Hours time difference
 How did we scale?
Romania
Bosnia
Ireland
Austria

14. Team setup & collaboration

15. © 2011 INFONOVA GmbH 15
Prereqesites scaling distributed team
 Optimum Teamsize: 5-7 incl. Scrum Master
 Every team has balanced skills – each team can work at least in 2-3
domains
 New teams get formed of existing staff, new team members get added to
already working teams
 Only experienced scrum masters should take over the building of new
teams
 Recruiting and personal changes determine the scaling speed

16. © 2011 INFONOVA GmbH 16
Scaling Teams
 Working Teams
Domain Experts
Scrum Master

17. © 2011 INFONOVA GmbH 17
Scaling Teams
 Extending Teams with new teamembers
 Teams get only extended with the right skilled people
Domain Experts
Scrum Master

18. © 2011 INFONOVA GmbH 18
Scaling Teams
 Shrink and grow
 If teams can also operate with four developers they are candidates fo be split up
to form a new team (yellow).
Domain Experts
Scrum Master

19. © 2011 INFONOVA GmbH 19
Scaling Teams
 Shrink and grow
 If teams can also operate with four developers they are candidates fo be split up
to form a new team (yellow).
Domain Experts
Scrum Master

20. © 2011 INFONOVA GmbH 20
Scaling Teams
 Shrink and grow
 If teams can also operate with four developers they are candidates fo be split up
to form a new team (yellow).
Domain Experts
Scrum Master

21. © 2011 INFONOVA GmbH 21
Scaling Teams in a distributed setting
 Enforce fully distributed teams
 direct communication between developers
 Distribute a team between at most two sites
 Number of newbies shall not exceed the number of experienced engineers
 Regular visits of team members between locations

22. © 2011 INFONOVA GmbH 22
Velocity Eircom
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10 11 12
Team
size
/
story
points
Sprint
Romania
Ireland
Austria Juniors
Austria Seniors
Velocity
Team size trend
Velocity trend

23. © 2011 INFONOVA GmbH 23
Velocity Eircom
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10 11 12
Team
size
/
story
points
Sprint
Romania
Ireland
Austria Juniors
Austria Seniors
Velocity
Team size trend
Velocity trend

24. © 2011 INFONOVA GmbH 24
Visualization and
automated Feedback
Lesson 2:

25. © 2011 INFONOVA GmbH 25
Feedback for agile Software Developers

26. © 2011 INFONOVA GmbH 26
 Did I break any other functionality?
 Is what I implemented understandable to the other developers? How is the
quality of the code I wrote?
 Are we able to deliver what I implemented?
Why Feedback?

27. © 2011 INFONOVA GmbH 27
Agile Development: Continuous Feedback
Pair-Programming
Unit-Tests
Continuous
Integration
Sek.
Min.
Code Reviews
Acceptance Tests
Static Code Analysis
Std./Tag

28. © 2011 INFONOVA GmbH 28
Continuous Integration Framework
2) SVN
Check Out
Developer
3) CI build & run
unit tests
1) SVN
Commit
4) Feedback
Artifact Repository
CI Server
Source Code Repository
Testsystem
5) Deploy, Run
Acceptance Tests
6) Feedback
SCA-System
8) Feedback
7) SCA Build

29. © 2011 INFONOVA GmbH 29
 Challenge: How to pair programm between different
sites?

30. © 2011 INFONOVA GmbH 30
 November: Starting with 5 databases, 1 continuous server, 1 delivery tool, 1
daily system, 1 demo system, 1 static code and test analytics server (Sonar).
 December: The traffic light visualization of unit test success.
 January: Additional 3 databases in Ireland, 1 new database in Romania.
 February: Additional integration test system with fixed testing team.
 March: Additional 2 databases in Romania and 6 databases in Ireland.
 April: Additional daily release system for release 1, early feedback for DB
syntax scripts.
Additional Infrustructure for distributed
development

31. © 2011 INFONOVA GmbH 31
Lesson 3: Common understanding of code
and code quality

32. © 2011 INFONOVA GmbH 32
Manual Code Reviews and Code Metrics

33. © 2011 INFONOVA GmbH 33
Establish common mindset about code quality
 Coding catas
 Bootcamps
 Reading groups
 Clean code development meetings
 Techboards
 DevCafes
 Pair programming
 Workplace learning

34. © 2011 INFONOVA GmbH 34
Useful contiuous improvement practices
Claudia Reithner
AssociateRecruiter
INFONOVA GmbH
Seering 6
8141 Unterpremstätten/Graz
Austria
Tel: +43 316 8003 8034
Fax: +43 316 8003 1481
Mobile: +43 664 81 61905
claudia.reithner@infonova.com
www.infonova.com