Test Driven
Development of
Embedded Systems
Marcin Czenko
Martijn Gelens
Wim van de Goor
Agile Testing Days, 14 october 20...
• Test Consultancy at VIIQ.
• Agile Test Team leader at Philips CareServant.
• 10 years of experience in the Testing Field...
• Currently, Philips Software Engineering Services,
MiPlaza, Software Designer.
• Ph.D. in Computer Security from Universi...
Wim van de Goor
• Agile Software Team Leader at Philips Software
Engineering Services, Philips MiPlaza.
• 8 years experien...
Contents
• Agile Testing
What Do we want to do?
• Constraints
What can we do?
• Agile Embedded Testing
How do we test?
• C...
What do we want ?
• Prerequisites
• Test Strategy
• Acceptance Criteria
• Continuous Integration
• Testing
Prerequisites
• Testing is integrated in the team's Way of
Working.
• Acceptance Criteria are defined, discussed and
explor...
Test Strategy
• Product risks and mitigation.
• Tester-role.
• Definition of Done.
• Quality gates.
• Testing is structured...
Test Strategy (cont.)
• Deliverables from outside (also HW).
• Issue procedure and attendees.
• Reporting (test reports, P...
Test Strategy (cont.)
The test strategy is build around the iterations and
hardware deliveries.
Acceptance Criteria
• Defined upfront
• Based upon Quality Attributes; functionality,
usability, efficiency, maintainability...
Continuous Integration
Automate your:
• Build procedure.
• Release package creation.
• Deployment to simulator.
• Deployme...
Continuous Integration
Continuous Integration
Test Driven Development
• Means: “Write unit tests
before code”.
• Integrate with your
Continuous Integration
environment....
This is your framework
Constraints
Light to Heavy
Light Heavy
Light
• No cross-compilation required.
• Usually mainstream OS (Windows/Linux).
• Wide range of testing/mocking
frameworks...
Heavy
• Small memory (no OS), limited performance,
limited debugging possibilities.
• Limited cross–compilers: often only ...
Hardware design
challenges
Using Evaluation Boards
Getting There (1)
Getting There (2)
Getting There (3)
Getting
There (4)
Getting There (5)
Getting There (6)
Getting There (7)
Development Board
• Selecting the right board can be
challenging (expensive ?).
• Chip selection driven by the availabilit...
Is there a more lean
solution ?
Are we lean ?
Queue 1
Queue 2
Queue 3
The effect on Agile
Principles
• Longer ramp-up time.
• Resistance to modify hardware
(introduces up-front design).
• Limi...
How to proceed ?
• Often we cannot remove queues &
batches in HW development (are we
going to be able doing so in any
pred...
Getting There (7)
Fix in between...
Fix in between...
How do we test ?
Our experiences
What do we need
• Testing Strategy.
• Hardware to work with.
• Tool chain (compiler).
• Testing Framework.
• Mocking Frame...
Compiler
ISO C++
ANSI C
Testing Framework
Keep it simple !
Do-It-Yourself !
Testing Framework (C)
• Many frameworks are simple ports of
the frameworks for PC-based
development.
• Increased stack con...
CMock
• Easy to understand. Easy to customise.
Lightweight.
• Comes with Supporting Ruby-based
Mocking Framework.
• Ready ...
Testing Frameworks (C++)
• Run-Time Type
Information
(RTTI).
• Exceptions.
• ISO C++ compiler
needed.
• Gnu or Microsoft
a...
Testing Frameworks (C++)
• We could not find a framework that
compiles on GreenHills and WindRiver
C++ compilers (forget IA...
yaffut
• Our choice for unit testing in C++.
• Just one header file.
• Not meant for embedded: needs RTTI,
and C++ exceptio...
Mocking
• We did not succeed in using existing
frameworks. Our best candidate
GoogleMock does not even compile and
it is q...
Mocking - way to go
• Think of your own framework.
• We needed three “evenings” to create
our own mocking framework.
Educate your customer
Educate your customer
Conclusions
• Agile in Heavy Embedded is a challenge: we cannot change it, but
we can understand it and try to reduce its ...
Conclusions
• Agile in Heavy Embedded is a challenge: we cannot change it, but
we can understand it and try to reduce its ...
Questions
Upcoming SlideShare
Loading in …5
×

Agile Testing Days

913 views
826 views

Published on

This is the presentation we gave in 2009 during Agile Testing Days in Berlin. Even though it is already more than 2 years old, many things we said during the talk are very valid today. Some things did not change at all.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
913
On SlideShare
0
From Embeds
0
Number of Embeds
49
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Agile Testing Days

  1. 1. Test Driven Development of Embedded Systems Marcin Czenko Martijn Gelens Wim van de Goor Agile Testing Days, 14 october 2009, Berlin
  2. 2. • Test Consultancy at VIIQ. • Agile Test Team leader at Philips CareServant. • 10 years of experience in the Testing Field (7 years V-Model, 3 years Agile). • Some experience with coding. • Bachelor degree in Information Technics. • Married to Jeanne, has a dog (Sasha) and 2 cats. Martijn Gelens
  3. 3. • Currently, Philips Software Engineering Services, MiPlaza, Software Designer. • Ph.D. in Computer Security from University of Twente, Enschede, The Netherlands. • M.Sc. in Software Engineering at Warsaw University of Technology. • 7 years of experience as an embedded software developer (SterKom (Poland) and freelancer). • Modelling Languages: my master thesis + one year internship at Institute National des Télécomunications (INT), Evry, Cedex, France. • Agile, test-invected since one year. • Married to Beata and also has two cats (Mufasa and Ursus). Marcin Czenko, Ph.D.
  4. 4. Wim van de Goor • Agile Software Team Leader at Philips Software Engineering Services, Philips MiPlaza. • 8 years experience with eXtreme Programming. • Agile mentor and Coach. • Advised us on Agile Principles.
  5. 5. Contents • Agile Testing What Do we want to do? • Constraints What can we do? • Agile Embedded Testing How do we test? • Conclusions
  6. 6. What do we want ? • Prerequisites • Test Strategy • Acceptance Criteria • Continuous Integration • Testing
  7. 7. Prerequisites • Testing is integrated in the team's Way of Working. • Acceptance Criteria are defined, discussed and explored beforehand. • Test Driven Development. • Continuous Build and Integration. • Testing is structured (specify, verify, report).
  8. 8. Test Strategy • Product risks and mitigation. • Tester-role. • Definition of Done. • Quality gates. • Testing is structured (specify, verify, report).
  9. 9. Test Strategy (cont.) • Deliverables from outside (also HW). • Issue procedure and attendees. • Reporting (test reports, PMI, ...). • Emergency scenarios.
  10. 10. Test Strategy (cont.) The test strategy is build around the iterations and hardware deliveries.
  11. 11. Acceptance Criteria • Defined upfront • Based upon Quality Attributes; functionality, usability, efficiency, maintainability, reliability, portability, etc... • Definition of Done: Tested and accepted, Code reviewed, documents written, yellow sticker on the note, ... • Quality Gate: Acceptance tests passed, smoke test passed, ...
  12. 12. Continuous Integration Automate your: • Build procedure. • Release package creation. • Deployment to simulator. • Deployment to Board Support Package.
  13. 13. Continuous Integration
  14. 14. Continuous Integration
  15. 15. Test Driven Development • Means: “Write unit tests before code”. • Integrate with your Continuous Integration environment. • Automate the Acceptance Tests.
  16. 16. This is your framework
  17. 17. Constraints
  18. 18. Light to Heavy Light Heavy
  19. 19. Light • No cross-compilation required. • Usually mainstream OS (Windows/Linux). • Wide range of testing/mocking frameworks available. • Standard hardware - no or very limited hardware level programming required. • No dependence on a particular vendor (supplier).
  20. 20. Heavy • Small memory (no OS), limited performance, limited debugging possibilities. • Limited cross–compilers: often only C, and Embedded (Extended) C++ available. • Vendor specific. • Difficult to find a testing/mocking framework. • Custom hardware (ramp-up).
  21. 21. Hardware design challenges
  22. 22. Using Evaluation Boards
  23. 23. Getting There (1)
  24. 24. Getting There (2)
  25. 25. Getting There (3)
  26. 26. Getting There (4)
  27. 27. Getting There (5)
  28. 28. Getting There (6)
  29. 29. Getting There (7)
  30. 30. Development Board • Selecting the right board can be challenging (expensive ?). • Chip selection driven by the availability of the right board. • The board selection driven by the availability of the BSP and OS.
  31. 31. Is there a more lean solution ?
  32. 32. Are we lean ?
  33. 33. Queue 1
  34. 34. Queue 2
  35. 35. Queue 3
  36. 36. The effect on Agile Principles • Longer ramp-up time. • Resistance to modify hardware (introduces up-front design). • Limited response to changes. • Higher risk.
  37. 37. How to proceed ? • Often we cannot remove queues & batches in HW development (are we going to be able doing so in any predictable future ?). • Reducing the queue size is also often not an option. • Is there a lean solution ?
  38. 38. Getting There (7)
  39. 39. Fix in between...
  40. 40. Fix in between...
  41. 41. How do we test ? Our experiences
  42. 42. What do we need • Testing Strategy. • Hardware to work with. • Tool chain (compiler). • Testing Framework. • Mocking Framework.
  43. 43. Compiler ISO C++ ANSI C
  44. 44. Testing Framework Keep it simple ! Do-It-Yourself !
  45. 45. Testing Framework (C) • Many frameworks are simple ports of the frameworks for PC-based development. • Increased stack consumption. • Dynamic memory allocation.
  46. 46. CMock • Easy to understand. Easy to customise. Lightweight. • Comes with Supporting Ruby-based Mocking Framework. • Ready For “Heavy Embedded” - tests executed in batches.
  47. 47. Testing Frameworks (C++) • Run-Time Type Information (RTTI). • Exceptions. • ISO C++ compiler needed. • Gnu or Microsoft are preferred.
  48. 48. Testing Frameworks (C++) • We could not find a framework that compiles on GreenHills and WindRiver C++ compilers (forget IAR Extended Embedded C++). • It was cheaper and more effective to come with your own simple testing frameworks.
  49. 49. yaffut • Our choice for unit testing in C++. • Just one header file. • Not meant for embedded: needs RTTI, and C++ exceptions. • Easy to understand and customise. We made an RTTI and Exception-free version.
  50. 50. Mocking • We did not succeed in using existing frameworks. Our best candidate GoogleMock does not even compile and it is quite complex. • Does it mean that no one is doing TDD on embedded ? Probably not.
  51. 51. Mocking - way to go • Think of your own framework. • We needed three “evenings” to create our own mocking framework.
  52. 52. Educate your customer
  53. 53. Educate your customer
  54. 54. Conclusions • Agile in Heavy Embedded is a challenge: we cannot change it, but we can understand it and try to reduce its impact on agile software development. • The tester should be experienced in working with hardware, perhaps even more than a developer. • There is no one way: what you can do depends on the constraints you have (e.g. light to heavy). • Hardware development is far from being lean - and there is not that much we can change. • Development and support tools are far behind the needs of the agile teams. • Let your agile testing framework grow with your code.
  55. 55. Conclusions • Agile in Heavy Embedded is a challenge: we cannot change it, but we can understand it and try to reduce its impact on agile software development. • The tester should be experienced in working with hardware, perhaps even more than a developer. • There is no one way: what you can do depends on the constraints you have (e.g. light to heavy). • Hardware development is far from being lean - and there is not that much we can change. • Development and support tools are far behind the needs of the agile teams. • Let your agile testing framework grow with your code. Be agile.
  56. 56. Questions

×