Channel 4 broadcasting company in Finland provides live streams of sporting events on the Internet (Ruutu.fi). The software development is done by agile principles but even more straightforward and quicker than normal agile projects. Tomi Kaleva says they have changed the entire production environment and renewed all the mobile apps in the past year. As a result, the normal agile development speed wasn’t enough. The fast software development cycle makes software testing challenging as there isn’t sufficient time for test planning and testing execution. The solution was to precisely prioritize the testing, mostly ad hoc without test planning in advance. It was critical that the whole development team participate in the testing and that the software developers be ready to fix bugs. Since some tests were done by customers after the software was released to production, it was crucial to listen to customers’ feedback and to react quickly to repair problems. Super-agile is an effective process to quickly release software to market but the high risk of poor quality must be addressed.

1. AT16
Session
6/9/16 3:00 PM
Testing in a Super-Agile Software
Development Environment
Presented by:
Tomi Kaleva
Nelonen Media
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.techwell.com/

2. Tomi Kaleva
Nelonen Media
With more than twenty years of software testing experience, Tomi Kaleva is an
expert in agile development and testing processes. He is working in Television
Company Channel 4 as a production manager, responsible for the quality
assurance process of Ruutu.fi live streaming services. With knowledge in
manufacturing, media, and public administration industries, Tomi has been a
software tester, test manager, test consultant, and quality department manager.
He has worked with dozens of organizations to assess the quality of their
software. Tomi has participated as speaker and chairman at several software
conferences. Learn more about Tomi at LinkedIn.

3. 5/30/16
1
Tes$ng in a Super-Agile So2ware
Development Environment
9.6.2016
Super-Agile Development
§ Presenta>on of Nelonen (Channel 4) and Ruutu (Screen)
§ What were the challenges of development?
§ How the development is organised?
§ What is “Super-Agile” development?
§ How the produc>on updates are done?
§ Tes>ng in Super-Agile development
§ How the bugs are fixed?
§ Is the ad-hoc tes>ng the solu>on?
§ Who leads developers or produc>on?
§ How does the Super-Agile work?
§ What next?
§ Ques>ons

4. 5/30/16
2
Strong and diverse media selec>on finds easily small and
larger target audiences
Weekly reach
480 000.
Targe$ng women
aged 25-44yrs.
Weekly reach
700 000.
Targe$ng men
aged 25-44yrs.
Weekly reach
1,2 million.
Most listened
radio sta$on within
25-44yrs.
Finland´s largest
digital network.
Weekly reach
2,5
million.
Weekly reach
2,6
million.
Weekly reach
3,6
million.
Million different
weekly users.
Greatest hits and
classics for the
young and young at
heart in Helsinki
metropolitan area.
Weekly reach
460 000.
Targe$ng women
aged 15-34yrs.
Pop music channel
targeted for
Helsinki
metropolitan area.
Weekly reach
900 000
million.
Approximately
2,4 million
video starts per week
Television on the screen.
For light users of
television, people looking
for something new or
those who missed a
programme.
Weekly reach
1 000 000
150 h
of new programming
every week
The Voice of Finland Suomen paras leipomo
Loirinuo8olla

5. 5/30/16
3
What were the challenges of development?
§ 5 years deal for streaming of Finnish Champion League ice-hockey matches
§ Old streaming player
§ New business needs
§ New applica>ons (iOS, Android, Windows Phone, SmartTV, Chromecast, …)
§ New architecture concept
§ New development partners
§ New quickly changed digital world
Time
”Watch what
is offered"
”Watch what ever,
when ever, where ever"
Main TV channels
Other TV channels
New ways to watch
Time used for TV watching
Habits of watching TV are changing quickly

6. 5/30/16
4
Developers
Ruutu backend
database videos
Mul>user Development Environment
§ The principle of Nelonen: ”all development and opera>on are
outsourced”
§ Nelonen team only coordinates and leads the development
§ All development companies belongs to one team, without
borderlines, “Nelonen team”
§ Version controls are opened to all developers
§ One common documenta>on
§ Development is also done over own borderlines
Ruutu

7. 5/30/16
5
Ruutu’s Roadmap
§ Planning 6 months ahead
§ Headlines of development for
each month
§ Sub>tles are detailed plans
§ Business is involved to
development planning
§ More issues at the begin of
table, more space at the end
”Super-Agile development”
§ Target is to get quickly out a working soeware
§ Straight communica>on, not through documenta>on
§ Quickly react to changes
§ Short itera>on rounds
§ Aeer each sprint there should be a ready working soeware
§ All teams work physically in same workspace
§ Subcontractor belongs to same team
§ Planning is done con>nuously, the whole project >me
§ Refractoring is done regularly

8. 5/30/16
6
Roadmap
Sprint planning
2 week Development
Follow-up
Version
Control
Produc$on Tes$ng
Produc>on update process
Thursday
Sprint planning
Tes>ng
environment
update
Automa>c
scripts to tes>ng
environment
Tes>ng
Tes>ng Produc>on update
Wednesday Friday Saturday Sunday Monday Tuesday
Automa>c scripts
to produc>on
environment
Automa>c
tests
Produc>on
update
informa>on

9. 5/30/16
7
Tes>ng in Super-Agile development
§ Fast cycle forces to quick tes>ng
§ Part of the tes>ng remain to produc>on, “customer tes>ng”
§ Developers have responsibility to larger tes>ng
§ In-house only acceptance tes>ng
§ The scope of tes>ng is challenging
– Different plahorms (iOS, Android, SmartTV, Chromecast)
– Different browsers (IE, Firefox, Chrome,…)
– Different opera>ng systems
§ Tes>ng priori>sing is done against to customer profiles
Is the ad-hoc tes>ng the solu>on?
§ Fast schedule force to new tes>ng solu>ons
§ Do we have to do hasty decision?
§ No >me to good regression tes>ng, only >me to test
the new features
§ Who defines test cases?
– Tester - > comprehensively, more >me needed
– Developer -> nothing could go wrong
– Business -> is the new feature working?
§ Beta tes>ng with customer
Even small tes8ng is be>er than nothing

10. 5/30/16
8
How the bugs are fixed?
§ The problem is marked to backlog
§ Priori>sing – Priori>sing – Priori>sing
– Balle against to business needs
§ Bug fixing
– Are the developers responsible for tes>ng?
§ Tes>ng version
– Which is the latest version?
– Is this the last version before produc>on update?
§ Retes>ng
– Is there enough >me to regression tes>ng?
§ Produc>on update
– When is the suitable moment to update the produc>on?
Who leads developers or produc>on?
§ We can’t only fix bugs, we need also to
develop
§ It is exci>ng to develop new features
§ Let’s make a hohix
§ Business pressures
§ More complex environment (new plahorms,
new products)
§ Produc>on problem: “fixed and gone”
§ Documenta>on takes too much >me
§ Friday aeernoon – let’s make a produc>on
update…
§ When is the best >me for produc>on update?
§ Is the tes>ng done enough comprehensive?
§ Who has the responsibility of tes>ng?
Developer or Produc>on?
§ Pressure from customers
§ Let’s not blow up ones the whole package
§ New features will break old features
§ Where is the update log?
§ Tes>ng and produc>on cost too much
Developer Produc9on

11. 5/30/16
9
How does the Super-Agile work?
§ Super-Agile method is fast and flexible
§ Fast reac>on to changes
§ Business needs are easily fulfilled
§ No bureaucracy
§ No >me spend to documenta>on
§ Compact development team
§ Fast reac>on to customer feedbacks
§ Development is possible to execute in small
groups
§ No >me to tes>ng
§ No >me scheduled to tes>ng
§ Bugs are frequently found in produc>on
environment
§ Lack of documenta>on and changelog
§ Communica>on cri>cal, example Skype chat
§ The team should work closely, if possible in
same room
§ Group discipline, no possibility to solo person
§ Using several subcontracts
What next?
§ Development cycle and speed must slow down
§ Produc>on updates should be scheduled and
predicted
§ Let’s not blow ones the whole package
§ Project plan and schedule
§ Separated development, tes>ng and produc>on
environments
§ Outsourcing tes>ng?
§ Listening more customer feedbacks
§ Big company – own regula>ons
§ Are we at the end of agile development?
§ Is everything ready?

12. 5/30/16
10
Opera>on Manager
Tomi Kaleva
0400 984638
tomi.kaleva@nelonenmedia.fi
@kalevaT