The document discusses various testing frameworks and approaches including unit test frameworks, functional test frameworks, test-driven development, behavior driven development, and using plain functions without a framework. It also summarizes the goals and evolution of a test framework created for OpenStack including initial goals, challenges with the first version, and improvements in the second version. The document concludes with recommendations around modularity, practices over specific tools, reuse of existing libraries, ensuring tests are documented and packaged, allowing for experimentation, and not over-engineering solutions.

2. •
•
•
•
•
•
•

5. UNIT TEST FRAMEWORK?
FUNCTIONAL TEST FRAMEWORK?
WHAT ABOUT BEHAVIOR DRIVEN DEVELOPMENT?
TEST DRIVEN DEVELOPMENT?
CAT DRIVEN DEVELOPMENT?
WHAT ABOUT DEPENDENT AND ORDERED TESTS?
WHO NEEDS TEST FRAMEWORKS!? JUST USE FUNCTIONS AND
WRITE EVERYTHING IN-LINE!

6. * In Daryl’s opinion. Subject to change based on future Daryl’s new information.

8. Create a test framework for OpenStack
Must be developed in Python
Must use open source libraries and tools
Should be capable of testing both JSON and XML
request/response content
Proof of Concept developed internally, but maintained in
the long term by the community
Initial focus was Compute, but should be able to be used to
test any OpenStack project

9. •
•
•
•

10. Python
unittest
Base Test Fixtures
pytest nosetest
httplib2
Zodiac OpenStack
Language Bindings
Paramiko
Linux Instance Validator
ConfigParser
Config Models

13. No prescribed
Python test runner
One monolithic
project
Single hard coded
configuration file
Poor logging
Credentials were
stored in the clear

15. Design a repeatable architecture for test and test
infrastructure
Initial focus was testing OpenStack, but should be able to
be used to test all internal projects at Rackspace
Must be developed in Python
Must support multiple test runners
Should be extendable

16. Python
Test Runners
unittest pytest behave
App Drivers
Requests
(HTTP)
HTTPClient
AutoMarshalling
HTTPClient
Paramiko (SSH)
LinuxClient
pywinrm
(WinRM)
WindowsClient
Config Manager
Files Env Variables
Remote Instance Validation Client

18. Broke the single repository
pattern
Provided foundational
“drivers” for different types
of tests
Provided a pattern for
other teams to repeat with
their own test projects
Created a foundation for
discussions on future API
testing practices

19. Composites
Cafe Projects

20. Creating confusing
naming conventions
Being prescriptive
when not necessary
Some components
features exceeded
by alternatives
Serialization code
was tedious
Lack of
documentation and
self testing
You can extend the
framework…in any
way we tell you to

24. Test runner A test model Assertions
Means of interacting
with the application
under test
Handling of test
data
Handling of test
results

26. cafe-runner and
cafe-parallel
unittest.TestCase unittest
cafe-http-plugin
cafe.configurator.
TestEnvManager
cclogging and
cafe.reporting

27. cafe-runner and
cafe-parallel
unittest.TestCase unittest
cafe-http-plugin
cafe.configurator.
TestEnvManager
cclogging and
cafe.reporting
Test Runner Test Model Assertions
Application Driver Test Data
Management
Test Artifact
Management

28. pytest unittest.TestCase Pyvows.expect
behest conflagration
Standard Python
logger and PyTest
result formatter
Test Runner Test Model Assertions
Application Driver Test Data
Management
Test Artifact
Management

29. Think modularly in
your test solutions
1
Encourage practices
over specific tools or
libraries
2
Don’t re-invent the
wheel (unless there
is no wheel)
3
That which is not
tested, documented,
and packaged does
not exist
4
Leave room for
experimentation
5
If you try to come
up with a solution
for everything, you
may end up pleasing
no one
6

32. •
•
•
•