stubs that also check that they are used correctly
Mocks: objects with expectations on how they should be used
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 27 / 47
Mocking with ScalaMock
ScalaMock: mocking framework for Scala
1. Define a mock for the interface/trait to mock
```scala
val mock = mock[MyInterface]
```
2. Set expectations on the mock
```scala
(mock.doSomething _).expects("arg").returning("result")
```
3. Pass mock to code under test
4. Verify
2. Outline
Outline
Recap on unit tests in JUnit and TDD
Operative introduction to “automated builds with tests and
dependencies” (in sbt)
Main kinds of tests: unit, integration, functional, stress, acceptance
Testing framework for Scala: ScalaTest
Test coverage
Introduction to test doubles and mocking
Some pointers to more advanced stuff on testing
Effective unit testing
Property-based testing (cf., QuickCheck, ScalaCheck)
Cucumber for behavior-driven development
Selenium for testing web apps via browser automation
Testing of GUI-based programs
Performance regression testing with ScalaMeter
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 2 / 47
3. Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 3 / 47
4. Recap: JUnit
1 public class DeviceImplTest {
2 private DeviceImpl device;
3
4 @Before public void init (){
5 this.device = new DeviceImpl ();
6 }
7
8 @Test public void on() throws Exception {
9 this.device.on();
10 assertTrue(this.device.isOn ());
11 }
12 // ..........
13 }
Concepts
Test suite, test cases, tests
Assertions
Fixture
Arrange, Act, Assert
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 4 / 47
5. Recap: Test-Driven Development (TDD)
What
“Development” is guided (“driven”) by “tests”
I.e., it is not a testing activity; tests as (useful) side-effect
Exercise your code before even writing it
Makes you explore the problem before attempting a solution
Forces you to think as the user of your code
Makes you focus on what is important right now (cf., KISS)
Red – Green – Refactor cycle
Benefits
Specification before implementation
Guides the (detailed) design process
You end up with regression tests (i.e., tests to catch regressions)
A regression is when something correct gets broken
A safety net against regressions is fundamental for refactoring activities
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 5 / 47
6. Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 6 / 47
7. The Scala Build Tool (sbt): intro
Goal: improve productivity through automation of project-related activities
sbt: the “standard” build tool for Scala
Belongs to the family of build tools, like Maven, Gradle and others
Gradle is the one that will be covered in this course
Not limited to Scala projects
Can seamlessly be used for Java projects
Plugins for other languages (e.g., Groovy, Clojure, C++a
, Jythonb
..)
a
https://github.com/d40cht/sbt-cpp
b
https://github.com/markchadwick/jython-sbt-plugin
Lightbend Activator
A custom version of sbt which adds two extra commands, activator
ui and activator new
Supports bootstrapping applications from a catalog of templates
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 7 / 47
8. The Scala Build Tool (sbt): basics
Key concepts
A build definition is defined in /build.sbt
It specifies a set of projects
Each projects is associated with a set of settings and tasks
Two modes of operation
1. Batch mode: $ sbt clean compile
2. Interactive mode
1 $ sbt
2 sbt > test
3 ...
4 sbt > console
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 8 / 47
10. The Scala Build Tool (sbt): basic usage
Basic commands
help: shows help
settings: lists the settings for the current project
tasks: lists the tasks for the current project
<settingName>: shows the value of the setting
<taskName> ..args..: execute the task
Common tasks
clean: deletes files produced by build
compile: compiles sources
console: starts Scala interpreter with project classes in classpath
package: produces a main artifact (e.g., a JAR)
runMain <mainClass> ..args..
test: executes all tests
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 10 / 47
11. Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 11 / 47
12. Perspectives on testing1
“Schools” of testing
Analytic School: testing as a rigorous task, via formal methods
Motto: Programs are logical artifacts, subject to math laws
Factory School: testing to measure project progress
Motto: Testing must be planned/scheduled/managed
Quality School: testing as a quality process
Motto: A quality process for a quality product
Context-Driven School: testing as a stakeholder-oriented, adaptable
process
Key question: which testing would be more valuable right now?
Agile School: testing to facilitate change
Techniques: test automation, test-driven approaches
Key takeaway: perspectives on testing impact the what and how of testing
1
https://www.prismnet.com/~wazmo/papers/four_schools.pdf
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 12 / 47
13. Many different kinds of testing
Multiple dimensions
Technology-facing vs. Business-facing testing
Verification vs. validation
Granularity/scope of testing
Unit – integration – system-level/end-to-end/acceptance
Static vs. dynamic
Code-level vs. runtime
Manual (human-based) vs. automated (computer-based)
Examples of human testing: inspection (code reviews), walkthrough
Examples of computer-based testing: static analysis, “standard” testing
Structural (aka white-box) vs. functional (aka black-box)
Examples of structural testing: control flow testing, path testing
Examples of functional testing: equivalence partitioning, BVA
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 13 / 47
14. Levels of testing
Levels of testing based on granularity/scope
1. Unit testing: testing at the level of individual units (of functionality)
2. Integration testing: testing of the functionality provided by multiple
integrated/interacting units
3. System testing: testing the whole system for correctness
In a black-box way
Non-functional requirements (e.g., efficiency, reliability, security) are
typically tested at this level
4. Acceptance testing: testing the whole system against the needs and
expectations of users and stakeholders
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 14 / 47
15. Acceptance tests
Acceptance testing
Is the SW system “acceptable” (for delivery)?
It’s about testing whether the SW system satisfies the acceptance
criteria (e.g., as specified in the requirements)
Acceptability is matter of customers, product owners, stakeholders, or
users
It is very much about what has to be built
⇒ Acceptance test-driven planning/development (ATDP/D)
Tools: FitNesse (wiki-based), JBehave, Cucumber (see later)
Acceptance testing vs. system testing
System tests ⇒ build the thing right (things should behave correctly)
Acceptance tests ⇒ build the right thing (validate what you want)
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 15 / 47
16. Scalability testing
Scalability testing: how is the ability of the system to accomodate
increasing loads?
Stress testing
Testing the system under unusual conditions (e.g., DoS attack)
Testing the upper limits (e.g., by exceeding the max load or
diminishing resources) until the SW crashes
Strictly related to reliability and availability requirements
Load testing
Testing system performance under high-load conditions
Example metrics: maximum/average response time, throughput
Tools: Gatlinga, The Grinderb,
a
http://gatling.io/
b
http://grinder.sourceforge.net/
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 16 / 47
17. Code/Test coverage
Coverage
Several definitions/metrics (and inconsistent terminology)
Many “things” can be “covered”, and at many levels
Code coverage: which/how much code is executed during testing
Mainly used for finding untested codea
Note: 100% coverage doesn’t mean that tests cover all the scenarios
Test coverage: how much of the behaviour is tested
The goal is not 100% coverage → testing the “right things”
a
https://martinfowler.com/bliki/TestCoverage.html
Further readings
“Coverage is not strongly correlated with test suite effectiveness” (Inozemtseva and
Holmes 2014) [3]
“How to misuse code coverage” (Marick 1999) [5]
“A survey of coverage-based testing tools” (Yang, Li, and Weiss 2009) [9]
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 17 / 47
19. Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 19 / 47
20. Testing in Scala: ScalaTest
Key features
Provides integration with many tools to provide a full, seamless
testing experience
JUnit/TestNG, ScalaCheck, JMock, ScalaMock, ...
Ant, Maven, sbt
Eclipse, NetBeans, IntelliJ
Supports several styles of testing out of the box
Recommendation: 1 style for unit testing, 1 style for acceptance testing
Configuration in sbt:
1 val scalatest = "org.scalatest" % " scalatest_2 .12" % "3.0.1" % "
test"
2
3 val commonsettings = Seq( scalaVersion := "2.12.2")
4
5 lazy val root = (project in file(".")).
6 settings( commonsettings ).
7 settings(name := "hello -scalatest",
8 libraryDependencies ++= Seq(scalatest))
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 20 / 47
21. ScalaTest: styles I
http://www.scalatest.org/user_guide/selecting_a_style
FunSuite: test functions as in classical XUnit libraries
1 class SetSuite extends FunSuite {
2 test("An empty Set should have size 0") {
3 assert(Set.empty.size == 0)
4 }
5
6 test("Invoking head on an empty Set should produce
NoSuchElementException ") {
7 assertThrows [ NoSuchElementException ] {
8 Set.empty.head
9 }
10 }
11 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 21 / 47
22. ScalaTest: styles II
FlatSpec: flat specifications `a la BDD
BDD (Behaviour-Driven Development) is a refinement of (A)TDD
From “tests” to “behaviour specifications”
Emphasis on communication
Stylistic impact + tooling impact (cf., Cucumber @ next slide)
1 class SetSpec extends FlatSpec {
2 "An empty Set" should "have size 0" in {
3 assert(Set.empty.size == 0)
4 }
5
6 it should "raise NoSuchElementException when head is invoked" in
{
7 assertThrows [ NoSuchElementException ] {
8 Set.empty.head
9 }
10 }
11 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 22 / 47
24. ScalaTest: styles IV
Many styles: recap
FunSuite: for xUnit people
FlatSpec: transition from xUnit to BDD-style
FunSpec: for BDD people
... other variants, e.g., giving more flexibility in the BDD style, ...
PropSpec: for property-based testing (see @ next slide)
FeatureSpec: for acceptance BDD (see @ next slide)
Guidelines for choice
One main style for unit testing (e.g., FlatSpec)
One main style for acceptance testing (e.g., FeatureSpec)
PropSpec when property-based testing suited
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 24 / 47
25. Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 25 / 47
26. Test doubles [6]: why and what
Test doubles: objects to be substituted for the real implementation for
testing purposes
Allow to gain full control if the context/environment of a piece of
code
Test doubles are essential for good test automation
Allowing isolation of code under test (from its dependencies)
Speeding-up test execution
Making random behavior deterministic
Simulating particular conditions that would be difficult to create
Observing state & interaction otherwise invisible
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 26 / 47
27. Kinds of test doubles
Kinds of test doubles
Dummy: objects passed around but never actually used; often used
to fill parameter lists (e.g., as null)
Stubs (“unusually short things”): simple implementations, e.g., with
methods that do nothing or return default values
Fake objects: have working implementations but provide a
fast/simple alternative (e.g., in-memory DB)
Test spies: stubs that record information about how they are called
Mocks: test spies configured/prescribed to behave in a certain way
under certain circumstances
Tests also verify mocks are used in a certain way (white-box)
References
http://xunitpatterns.com/TestDouble.html
https://martinfowler.com/articles/mocksArentStubs.html
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 27 / 47
28. Mocking with ScalaMock I
Usage
1 libraryDependencies += "org.scalamock" %% "scalamock -scalatest -support" % "3.5.0" % Test
ScalaMock can be used in tests written in ScalaTest (or Specs2)
In ScalaTest, mix trait MockFactory in your test class
Mocking styles in ScalaMock
Expectations-First style: expectations are set on mocks before
exercising the System Under Test (SUT)
If these expectations are not met, the test fails
Record-then-Verify style: expectations are verified on stubs after
the SUT has executed
In ScalaMock, this is supported by stubs
(1) for objects for which we have strict expectations, (2) otherwise
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 28 / 47
29. Mocking with ScalaMock II
SUT
1 case class Request(body: String)
2 case class Response(content: String)
3
4 class Server {
5 def processRequest (req: Request): Response = Response(req.body+ "’s Response")
6 def serve(req: Request): Response = processRequest (req)
7 }
8
9 trait Cache[K,T] { // Note: we provide no implementation class yet
10 def cached(key: K): Boolean
11 def get(key: K): T
12 def put(key: K, value: T): Unit
13 }
14
15 class ServerWithCache (private val cache: Cache[Request , Response],
16 private val cacheable: Request => Boolean) extends Server {
17 override def serve(req: Request) = if(! cacheable(req)) super.serve(req) else {
18 if (cache.cached(req)) {
19 cache.get(req)
20 } else {
21 val res = processRequest (req)
22 cache.put(req , res)
23 res
24 }
25 }
26 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 29 / 47
30. Mocking with ScalaMock III
1 class TestWithMock extends FunSuite with MockFactory with Matchers {
2 test("Test server with cache"){
3 // Mocks/stubs for dependencies
4 val cacheMock = mock[Cache[Request ,Response ]]
5 val cacheableStub = stubFunction [Request , Boolean]
6 cacheableStub .when (*).returns(true)
7
8 // Wire SUT with stubbed/mocked dependencies
9 val server = new ServerWithCache (cacheMock , cacheableStub )
10
11 // Arrange
12 val request = Request("Some request")
13 val expectedResponse = Response(request.body+"’s Response")
14
15 // Mock expectations
16 inSequence {
17 (cacheMock.cached _).expects(request).returning(false)
18 (cacheMock.put _).expects(request , *)
19 (cacheMock.cached _).expects(request).returning(true)
20 (cacheMock.get _).expects(request).returning( expectedResponse )
21 }
22
23 // Act + Assert
24 server.serve(request) shouldEqual expectedResponse
25 server.serve(request) shouldEqual expectedResponse
26 cacheableStub .verify(request).twice () // (cf., test spy)
27 }
28 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 30 / 47
31. Outline
1 Quick recap
2 sbt
3 Panorama on testing
4 Basics of testing in Scala with ScalaTest
5 Test doubles
6 More on testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 31 / 47
32. On effective unit testing
Key point: (software) quality should also be applied to testing code
3 key properties: readability, maintainability, trustworthiness
Good unit tests should
be automated and repeatable
be easy to implement
be relevant tomorrow
be runnable by anyone with ease
run quickly
be consistent in its results
have full control of the unit under test
be fully isolated
be communicative when failing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 32 / 47
33. On testable design
Testability of software components
Refers to how much effort is needed for testing
Wrt a certain testing technique
Factors affecting testability include
Observability
Controllability
Number of dependencies and how they are resolved
Number of responsibilities
Principles of testable design
Modularity
SOLID (SRP, OCP, LSP, ISP, DIP)
Inversion of Control
Cf., Dependency Injection (note: not “Dependency Inversion”)
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 33 / 47
34. Property-based testing with ScalaCheck (i)
Property-based testing: key idea
Decoupling specification of behaviour from creation of test cases
1) You express properties that must hold for certain types of data
2) Tools automatically generate test cases trying to falsify those props
When a prop is falsified, the (minimal) counterexample is given
ScalaCheck
Key abstractions
org.scalacheck.Prop: a property, the testable unit in ScalaCheck
Kinds: universally quantified, existentially quantified, conditional, ..
Combinators to compose props, e.g., p && q, atLeastOne(p,q,r)
org.scalacheck.Gen: a generator, for producing test data
Combinators for creating custom generators
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 34 / 47
36. ScalaTest: property-based testing support
PropSpec: for property-based test suites
Two styles for property checks:
1. ScalaTest style: mix PropertyChecks trait in
2. ScalaCheck style: mix Checkers trait in
1 class SetSpec extends PropSpec with PropertyChecks with Checkers with Matchers {
2 val examples = Table("set",BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int])
3 val examplesGen = Gen.oneOf(BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int])
4
5 property("an empty Set should have size 0") {
6 // ScalaTest ’s property -based testing style
7 forAll(examples) { set => set.size should be (0) }
8 // ScalaCheck ’s property -based testing style
9 check { Prop.forAll( examplesGen ) { set => set.size == 0 } }
10 }
11
12 property("invoking head on an empty set should produce NoSuchElementException ") {
13 // ScalaTest ’s property -based testing style
14 forAll(examples) { set => a [ NoSuchElementException ] should be thrownBy{set.head} }
15 // ScalaCheck ’s property -based testing style
16 check { Prop.forAll( examplesGen ) { set =>
17 Prop.throws(classOf[ NoSuchElementException ])(set.head) }
18 }
19 }
20 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 36 / 47
37. Behaviour-Driven Development (BDD) with Cucumber (i)
Motivation
Remember: acceptance testing = are we building the right thing?
Agile approach: iterative, incremental development and frequent
interactions with stakeholders to get feedback
But requirements need to be communicated, different stakeholders
may use their particular jargon, ..
Behaviour-Driven Development (BDD)
⇒ Idea: collaborative specification of acceptance tests
≈ Acceptance Test-Driven Dev. (ATDD) + customer-orientation
Acceptance tests as executable specs and a living documentation
Pros: ubiquitous language (cf., DDD) + “definitive source of truth”
Acceptance tests specified as examples of the way we want the
system to behave in particular scenarios
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 37 / 47
38. Behaviour-Driven Development (BDD) with Cucumber (ii)
Cucumber: basics
Features are high-level specs expressed by a user-perspective in a
English-like DSL called Gherkin
Features consist of multiple scenarios, representing concrete
examples/use cases
Scenarios are structured into multiple steps (given, when, then)
sbt configuration
1 val commons = Seq( scalaVersion := "2.11.8")
2 val cuke_runner = "com.waioeka.sbt" %% "cucumber -runner" % "0.0.8"
3 val cuke_scala = "info.cukes" %% "cucumber -scala" % "1.2.4" % Test
4 val cuke_junit = "info.cukes" % "cucumber -junit" % "1.2.4" % Test
5
6 lazy val root = (project in file(".")).settings(commons).settings(
7 name := "hello -scala -cucumber",
8 libraryDependencies ++= Seq(cuke_runner ,cuke_scala ,cuke_junit))
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 38 / 47
39. Behaviour-Driven Development (BDD) with Cucumber (iii)
src/test/resources/features/MyFeature.feature
1 Feature: Numbers
2
3 Scenario: I want to multiply two nums
4 Given one operand 7
5 And another operand 8
6 When I multiply them together
7 Then I should obtain result 56
src/test/scala/RunCucumberTests.scala
1 @RunWith(classOf[Cucumber ])
2 @CucumberOptions (features = Array("classpath:features/MyFeature.feature"),
3 tags = Array("~@Wip"), glue = Array("classpath:steps"),
4 plugin = Array("pretty", "html:target/cucumber/html"))
5 class RunCucumberTests
src/test/scala/steps/CucumberSteps.scala
1 class CucumberSteps extends ScalaDsl with EN {
2 var (a, b, sum) = (0, 0, 0)
3
4 Given("""^one operand (d+)$"""){ a = (_:Int) }
5 Given("""^another operand (d+)$"""){ b = (_:Int) }
6 When("""^I multiply them together$"""){ sum = a*b }
7 Then("""^I should obtain result (d+)$"""){ (expectedSum :Int) =>
8 Assert. assertEquals (expectedSum , sum)
9 }
10 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 39 / 47
40. ScalaTest: acceptance testing style
FeatureSpec: for acceptance testing suites
1 class TVSet {
2 private var on: Boolean = false
3 def isOn: Boolean = on
4 def pressPowerButton () { on = !on }
5 }
6
7 class TVSetSpec extends FeatureSpec with GivenWhenThen {
8 info("As a TV set owner")
9 info("I want to be able to turn the TV on and off")
10 info("So I can watch TV when I want")
11 info("And save energy when I’m not watching TV")
12
13 feature("TV power button") {
14 scenario("User presses power button when TV is off") {
15 Given("a TV set that is switched off")
16 val tv = new TVSet
17 assert (!tv.isOn)
18
19 When("the power button is pressed")
20 tv. pressPowerButton ()
21
22 Then("the TV should switch on")
23 assert(tv.isOn)
24 }
25 }
26
27 scenario("User presses power button when TV is on") { ??? }
28 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 40 / 47
41. Web application testing with Selenium
Testing web applications
Selenium is a tool that provides support for browser-based tests
ScalaTest includes a DSL to drive Selenium tests
Ref.: http://www.scalatest.org/user_guide/using_selenium
Sbt dep: "org.seleniumhq.selenium" % "selenium-java" % "2.35.0" % "test"
1 class GoogletestSpec extends FlatSpec with Matchers with HtmlUnit {
2 "Google search" should "work" in {
3 go to "http :// www.google.com"
4 pageTitle should be ("Google")
5 executeScript ("return document.title") shouldEqual pageTitle
6
7 click on "q" // By name of the field
8 textField("q").value = "selenium"
9 submit ()
10
11 eventually(timeout (2 seconds)) {
12 pageTitle should startWith ("selenium - ")
13 }
14 }
15 }R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 41 / 47
42. Testing GUI-based programs
Key question
What does it mean to test a GUI-based program?
Answer “what should I test” before “how”
Approaches
1. Test monkey
2. Test monkey + logging
3. Mouse/Keyboard capture & replay tools
Scripts
4. Programmatic access to GUI
5. Image recognition to identify/control GUI components
E.g.: SikuliXa
a
http://sikulix.com/
https://en.wikipedia.org/wiki/Comparison_of_GUI_testing_tools
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 42 / 47
43. Microbenchmarking with ScalaMeter (i)
A microbenchmark is a measurement of the running time of some
isolated piece of code
Issues in doing microbenchmarking
Microbenchmark conditions/inputs are hard to reproduce
Performance measurement in general introduces observer bias
In general, runtime behaviour is nondeterministic
Inaccuracies in performance measurements can be due to
JIT-compilation, classloading, automatic memory management (GC), ..
Performance metrics (such as running time) inherently give an
incomplete picture about the performance characteristics
However, microbenchmarking may still be useful, e.g.,
for verifying that an optimisation is an optimisation at all
for comparison of different implementations wrt specific conditions
for supporting performance regression testing
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 43 / 47
44. Microbenchmarking with ScalaMeter (ii)
1 object Mapper {
2 def map1[T](xs: List[T])(f: T=>T): List[T] = xs.map(f)
3 def map2[T](xs: List[T])(f: T=>T): List[T] = xs.reverse.reverse.map(f)
4 }
5
6 import org.scalameter.api._ // ... other imports
7
8 object RegressionTest extends Bench. OnlineRegressionReport {
9 override val persistor: Persistor = SerializationPersistor ()
10 override val reporter: Reporter[Double] = Reporter.Composite(
11 new RegressionReporter (tester ,historian), HtmlReporter (! online))
12
13 val sizes = Gen.range("size")(1000 , 10000 , 3000)
14 val lists = for (sz <- sizes) yield (0 until sz).toList
15
16 performance of "List" in {
17 measure method "map" in {
18 using(lists) config (
19 exec.benchRuns -> 20, exec. independentSamples -> 1
20 ) in { xs =>
21 Mapper.map1(xs)(_+1)
22 }
23 }
24 }
25 }
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 44 / 47
45. References and further readings
Effective unit testing
Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich, CT,
USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276
L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series.
Manning, 2013. isbn: 9781935182573
Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper Saddle River, NJ,
USA: Prentice Hall PTR, 2006. isbn: 0131495054
On specific topics
Dorothy Graham and Mark Fewster. Experiences of Test Automation: Case Studies of
Software Test Automation. 1st. Addison-Wesley Professional, 2012. isbn: 0321754069,
9780321754066
Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st.
Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124
S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book: Behaviour-Driven
Development for Testers and Developers. Pragmatic programmers. Pragmatic Bookshelf,
2015. isbn: 9781941222294
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 45 / 47
46. Bibliography I
Dorothy Graham and Mark Fewster. Experiences of Test Automation: Case
Studies of Software Test Automation. 1st. Addison-Wesley Professional, 2012.
isbn: 0321754069, 9780321754066.
Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st.
Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124.
Laura Inozemtseva and Reid Holmes. “Coverage is not strongly correlated with
test suite effectiveness”. In: Proceedings of the 36th International Conference on
Software Engineering. ACM. 2014, pp. 435–445.
L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series.
Manning, 2013. isbn: 9781935182573.
Brian Marick et al. “How to misuse code coverage”. In: Proceedings of the 16th
Interational Conference on Testing Computer Software. 1999, pp. 16–18.
Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper Saddle
River, NJ, USA: Prentice Hall PTR, 2006. isbn: 0131495054.
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 46 / 47
47. Bibliography II
Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich,
CT, USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276.
S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book:
Behaviour-Driven Development for Testers and Developers. Pragmatic
programmers. Pragmatic Bookshelf, 2015. isbn: 9781941222294.
Qian Yang, J Jenny Li, and David M Weiss. “A survey of coverage-based testing
tools”. In: The Computer Journal 52.5 (2009), pp. 589–597.
R.Casadei (Universit`a di Bologna) Testing a.a. 2016/2017 47 / 47