Improving Correctness
with Types
Scala Days March 2015
Iain Hull
iain.hull@workday.com
@IainHull
http://workday.github.io
Defensive programming
Fail fast
Design by contract
“Bad programmers worry about the code.
Good programmers worry about
data structures and their relationships.”
- Linus Torvalds
What is a function ?
f
x y
Domain Range
Never use nulls … period
“Null references, my billion dollar mistake”
- Tony Hoare, QCon 2009
All data is immutable
Do not throw exceptions
Wrappers
Wrapper Types
case class Customer(name: String,
preferredCurrency: String) {
require(Currency.isValid(preferredCurrency))
}
val customer = Customer(name = "Joe Bloggs",
preferredCurrency = "SFO")
This is an airport?
class Currency private (val code: String) extends AnyVal
object Currency {
val USD: Currency = new Currency("USD")
val EUR: Currency = new Currency("EUR")
// ...
def from(code: String): Option[Currency] = ???
}
case class Customer(name: String,
preferredCurrency: Currency)
def creditCardRate(customer: Customer): BigDecimal = {
if (customer.preferredCurrency == "USD")
BigDecimal("0.015")
else
BigDecimal("0.025")
}
Always false
import org.scalactic.TypeCheckedTripleEquals._
case class Customer(name: String,
preferredCurrency: Currency)
def creditCardRate(customer: Customer): BigDecimal = {
if (customer.preferredCurrency === "USD")
BigDecimal("0.015")
else
BigDecimal("0.025")
}
Does not compile
import org.scalactic.TypeCheckedTripleEquals._
case class Customer(name: String,
preferredCurrency: Currency)
def creditCardRate(customer: Customer): BigDecimal = {
if (customer.preferredCurrency === Currency.USD)
BigDecimal("0.015")
else
BigDecimal("0.025")
}
val order: Order = ???
val customer: Customer = ???
val creditCardCharge = order.amount + creditCardRate(customer)
Eeek this a bug
class MoneyAmount(val amount: BigDecimal) extends AnyVal {
def + (rhs: MoneyAmount): MoneyAmount =
new MoneyAmount(amount + rhs.amount)
def - (rhs: MoneyAmount): MoneyAmount =
new MoneyAmount(amount - rhs.amount)
def * (rhs: Rate): MoneyAmount =
new MoneyAmount(amount * rhs.size)
}
class Rate(val size: BigDecimal) extends AnyVal {
def * (rhs: Rate): MoneyAmount = rhs * this
}
val order: Order = ???
val customer: Customer = ???
val creditCardCharge = order.amount + creditCardRate(customer)
Does not compile
Using wrapper types
• Do not abuse primitives
• Control the available values
• Control the available operations
• Move validation to the correct place
Non empty list
def average(items: List[Int]): Int = items.sum / items.size
scala> average(List(5))
res1: Int = 5
scala> average(List(5, 10, 15))
res2: Int = 10
scala> average(List())
java.lang.ArithmeticException: / by zero
at .average0(<console>:8)
... 35 elided
import org.scalactic.Every
def average(items: Every[Int]): Int = items.sum / items.size
scala> average(Every(5))
res1: Int = 5
scala> average(Every(5, 10, 15))
res2: Int = 10
scala> average(Every())
<console>:10: error: not enough arguments for method apply:
(firstElement: T, otherElements: T*)org.scalactic.Every[T] in
object Every.
Unspecified value parameters firstElement, otherElements.
average(Every())
import org.scalactic.Every
def average(items: Every[Int]): Int = items.sum / items.size
def average(first: Int, rest: Int*): Int =
average(Every(first, rest: _*))
scala> average(5)
res1: Int = 5
scala> average(5, 10, 15)
res2: Int = 10
scala> average()
<console>:11: error: not enough arguments for method average:
(first: Int, rest: Int*)Int.
Unspecified value parameters first, rest.
average()
Non empty lists
• Some lists cannot be empty
• Tell the compiler
• One-plus-var-args idiom
Algebraic data types
Agent Id Type Status Host In Use By
A01 1 Active 10.0.0.1
A02 1 Failed 10.0.0.2 J01
A03 2 Active 10.0.0.3 J03
A04 2 Waiting 10.0.0.4
Job Id Type Status Submitted By Processed By
J01 1 Waiting Fred
J02 1 Active Wilma A01
J03 2 Complete Barney A03
Jobs
Agents
case class Agent(agentId: String,
jobType: Int,
host: String,
port: Int,
status: String, // Waiting | Active | Failed
maybeLastAccessed: Option[DateTime],
inUse: Boolean,
maybeInUseBy: Option[String])
case class Job(referenceId: String,
jobType: Int,
status: String, // Waiting | Active | Complete
submittedBy: String,
submittedAt: DateTime,
maybeStartedAt: Option[DateTime],
maybeProcessedBy: Option[String],
maybeCompletedAt: Option[DateTime])
case class Agent(agentId: String,
jobType: JobType,
address: AgentAddress,
status: AgentStatus,
lastAccessed: Option[DateTime],
inUse: Boolean,
maybeInUseBy: Option[String])
case class Job(referenceId: String,
jobType: JobType,
status: JobStatus,
submittedBy: User,
submittedAt: DateTime,
maybeStartedAt: Option[DateTime],
maybeProcessedBy: Option[String],
maybeCompletedAt: Option[DateTime])
sealed abstract class JobType(val value: Int)
case object SmallJob extends JobType(1)
case object LargeJob extends JobType(2)
case object BatchJob extends JobType(3)
sealed abstract class AgentStatus(val value: String)
case object AgentWaiting extends AgentStatus("Waiting")
case object AgentActive extends AgentStatus("Active")
case object AgentFailed extends AgentStatus("Failed")
sealed abstract class JobStatus(val value: String)
case object JobWaiting extends JobStatus("Waiting")
case object JobActive extends JobStatus("Active")
case object JobCompelete extends JobStatus("Complete")
case class AgentAddress(host: String, port: Int)
case class User(name: String)
case class Agent(agentId: String,
jobType: JobType,
address: AgentAddress,
status: AgentStatus,
lastAccessed: Option[DateTime],
inUse: Boolean,
maybeInUseBy: Option[String])
case class Job(referenceId: String,
jobType: JobType,
status: JobStatus,
submittedBy: User,
submittedAt: DateTime,
maybeStartedAt: Option[DateTime],
maybeProcessedBy: Option[String],
maybeCompletedAt: Option[DateTime])
import tag.@@
trait Foo
def onlyFoo(value: String @@ Foo): String = s"It a foo: $value"
scala> onlyFoo("simple string")
<console>:13: error: type mismatch;
found : String("simple string")
required: tag.@@[String,Foo]
(which expands to) String with tag.Tagged[Foo]
onlyFoo("simple string")
^
scala> val foo = tag[Foo]("Foo String")
foo: tag.@@[String,Foo] = Foo String
scala> onlyFoo(foo)
res2: String = It a foo: Foo String
def anyString(value: String): String = s"Just a string: $value”
scala> anyString(foo)
res6: String = Just a string: Foo String
case class Agent(agentId: String @@ Agent,
jobType: JobType,
address: AgentAddress,
status: AgentStatus,
lastAccessed: Option[DateTime],
inUse: Boolean,
maybeInUseBy: Option[String @@ Job])
case class Job(referenceId: String @@ Job,
jobType: JobType,
status: JobStatus,
submittedBy: User,
submittedAt: DateTime,
maybeStartedAt: Option[DateTime],
maybeProcessedBy: Option[String @@ Agent],
maybeCompletedAt: Option[DateTime])
case class Job(referenceId: String @@ Agent,
jobType: JobType,
status: JobStatus,
submittedBy: User,
submittedAt: DateTime,
maybeStartedAt: Option[DateTime],
maybeProcessedBy: Option[String @@ Job],
maybeCompletedAt: Option[DateTime])
def recordCompletionMetrics(job: Job): Unit = {
for( startedAt <- job.maybeStartedAt ;
completedAt <- job.maybeCompletedAt ) {
writeJobEvent(
event = "Completed",
time = completedAt,
referenceId = job.referenceId,
waitingTime = (startedAt - job.submittedAt),
executionTime = (completedAt - startedAt))
}
}
def recordCompletionMetrics(job: Job): Unit = {
require(job.status = JobComplete)
require(job.maybeStartedAt.isDefined)
require(job.maybeCompletedAt.isDefined)
for (startedAt <- job.maybeStartedAt ;
completedAt <- job.maybeCompletedAt ) {
writeJobEvent (
event = "Completed",
time = completedAt,
referenceId = job.referenceId,
waitingTime = (startedAt - job.submittedAt),
executionTime = (completedAt - startedAt))
}
}
sealed trait Job {
def referenceId: String @@ Job
def status: JobStatus
def submittedBy: User
def submittedAt: DateTime
}
case class WaitingJob(referenceId: String @@ Job,
submittedBy: User,
submittedAt: DateTime)
extends Job {
val status: JobStatus = JobWaiting
}
sealed trait Job {
def referenceId: String @@ Job
def status: JobStatus
def submittedBy: User
def submittedAt: DateTime
}
case class ActiveJob(referenceId: String @@ Job,
submittedBy: User,
submittedAt: DateTime,
startedAt: DateTime,
processedBy: String @@ Agent)
extends Job {
val status: JobStatus = JobActive
}
sealed trait Job {
def referenceId: String @@ Job
def status: JobStatus
def submittedBy: User
def submittedAt: DateTime
}
case class CompleteJob(referenceId: String @@ Job,
submittedBy: User,
submittedAt: DateTime,
startedAt: DateTime,
processedBy: String @@ Agent,
completedAt: DateTime)
extends Job {
val status: JobStatus = JobComplete
}
def recordCompletionMetrics(job: CompleteJob): Unit = {
writeJobEvent(
event = "Completed",
time = job.completedAt,
referenceId = job.referenceId,
waitingTime = (job.startedAt - job.submittedAt),
executionTime = (job.completedAt - job.startedAt))
}
Algebraic data types
• Simply sealed traits and case classes
• Exposes the shape of your data
• Use this shape to control possible states
More advanced techniques
• Path dependent types
• Self-recursive types
• Phantom types
• Shapeless
Defensive
Programming
Fail Fast
Design by
Contract
Types
“A mind is like a parachute,
it doesn’t work unless its open”
- Frank Zappa
http://workday.github.io

Improving Correctness with Types

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
    “Bad programmers worryabout the code. Good programmers worry about data structures and their relationships.” - Linus Torvalds
  • 7.
    What is afunction ? f x y Domain Range
  • 9.
    Never use nulls… period “Null references, my billion dollar mistake” - Tony Hoare, QCon 2009
  • 10.
    All data isimmutable
  • 11.
    Do not throwexceptions
  • 12.
  • 13.
    case class Customer(name:String, preferredCurrency: String) { require(Currency.isValid(preferredCurrency)) } val customer = Customer(name = "Joe Bloggs", preferredCurrency = "SFO") This is an airport?
  • 14.
    class Currency private(val code: String) extends AnyVal object Currency { val USD: Currency = new Currency("USD") val EUR: Currency = new Currency("EUR") // ... def from(code: String): Option[Currency] = ??? }
  • 15.
    case class Customer(name:String, preferredCurrency: Currency) def creditCardRate(customer: Customer): BigDecimal = { if (customer.preferredCurrency == "USD") BigDecimal("0.015") else BigDecimal("0.025") } Always false
  • 16.
    import org.scalactic.TypeCheckedTripleEquals._ case classCustomer(name: String, preferredCurrency: Currency) def creditCardRate(customer: Customer): BigDecimal = { if (customer.preferredCurrency === "USD") BigDecimal("0.015") else BigDecimal("0.025") } Does not compile
  • 17.
    import org.scalactic.TypeCheckedTripleEquals._ case classCustomer(name: String, preferredCurrency: Currency) def creditCardRate(customer: Customer): BigDecimal = { if (customer.preferredCurrency === Currency.USD) BigDecimal("0.015") else BigDecimal("0.025") }
  • 18.
    val order: Order= ??? val customer: Customer = ??? val creditCardCharge = order.amount + creditCardRate(customer) Eeek this a bug
  • 19.
    class MoneyAmount(val amount:BigDecimal) extends AnyVal { def + (rhs: MoneyAmount): MoneyAmount = new MoneyAmount(amount + rhs.amount) def - (rhs: MoneyAmount): MoneyAmount = new MoneyAmount(amount - rhs.amount) def * (rhs: Rate): MoneyAmount = new MoneyAmount(amount * rhs.size) } class Rate(val size: BigDecimal) extends AnyVal { def * (rhs: Rate): MoneyAmount = rhs * this }
  • 20.
    val order: Order= ??? val customer: Customer = ??? val creditCardCharge = order.amount + creditCardRate(customer) Does not compile
  • 21.
    Using wrapper types •Do not abuse primitives • Control the available values • Control the available operations • Move validation to the correct place
  • 22.
  • 23.
    def average(items: List[Int]):Int = items.sum / items.size scala> average(List(5)) res1: Int = 5 scala> average(List(5, 10, 15)) res2: Int = 10 scala> average(List()) java.lang.ArithmeticException: / by zero at .average0(<console>:8) ... 35 elided
  • 24.
    import org.scalactic.Every def average(items:Every[Int]): Int = items.sum / items.size scala> average(Every(5)) res1: Int = 5 scala> average(Every(5, 10, 15)) res2: Int = 10 scala> average(Every()) <console>:10: error: not enough arguments for method apply: (firstElement: T, otherElements: T*)org.scalactic.Every[T] in object Every. Unspecified value parameters firstElement, otherElements. average(Every())
  • 25.
    import org.scalactic.Every def average(items:Every[Int]): Int = items.sum / items.size def average(first: Int, rest: Int*): Int = average(Every(first, rest: _*)) scala> average(5) res1: Int = 5 scala> average(5, 10, 15) res2: Int = 10 scala> average() <console>:11: error: not enough arguments for method average: (first: Int, rest: Int*)Int. Unspecified value parameters first, rest. average()
  • 26.
    Non empty lists •Some lists cannot be empty • Tell the compiler • One-plus-var-args idiom
  • 27.
  • 28.
    Agent Id TypeStatus Host In Use By A01 1 Active 10.0.0.1 A02 1 Failed 10.0.0.2 J01 A03 2 Active 10.0.0.3 J03 A04 2 Waiting 10.0.0.4 Job Id Type Status Submitted By Processed By J01 1 Waiting Fred J02 1 Active Wilma A01 J03 2 Complete Barney A03 Jobs Agents
  • 29.
    case class Agent(agentId:String, jobType: Int, host: String, port: Int, status: String, // Waiting | Active | Failed maybeLastAccessed: Option[DateTime], inUse: Boolean, maybeInUseBy: Option[String]) case class Job(referenceId: String, jobType: Int, status: String, // Waiting | Active | Complete submittedBy: String, submittedAt: DateTime, maybeStartedAt: Option[DateTime], maybeProcessedBy: Option[String], maybeCompletedAt: Option[DateTime])
  • 30.
    case class Agent(agentId:String, jobType: JobType, address: AgentAddress, status: AgentStatus, lastAccessed: Option[DateTime], inUse: Boolean, maybeInUseBy: Option[String]) case class Job(referenceId: String, jobType: JobType, status: JobStatus, submittedBy: User, submittedAt: DateTime, maybeStartedAt: Option[DateTime], maybeProcessedBy: Option[String], maybeCompletedAt: Option[DateTime])
  • 31.
    sealed abstract classJobType(val value: Int) case object SmallJob extends JobType(1) case object LargeJob extends JobType(2) case object BatchJob extends JobType(3) sealed abstract class AgentStatus(val value: String) case object AgentWaiting extends AgentStatus("Waiting") case object AgentActive extends AgentStatus("Active") case object AgentFailed extends AgentStatus("Failed") sealed abstract class JobStatus(val value: String) case object JobWaiting extends JobStatus("Waiting") case object JobActive extends JobStatus("Active") case object JobCompelete extends JobStatus("Complete") case class AgentAddress(host: String, port: Int) case class User(name: String)
  • 32.
    case class Agent(agentId:String, jobType: JobType, address: AgentAddress, status: AgentStatus, lastAccessed: Option[DateTime], inUse: Boolean, maybeInUseBy: Option[String]) case class Job(referenceId: String, jobType: JobType, status: JobStatus, submittedBy: User, submittedAt: DateTime, maybeStartedAt: Option[DateTime], maybeProcessedBy: Option[String], maybeCompletedAt: Option[DateTime])
  • 33.
    import tag.@@ trait Foo defonlyFoo(value: String @@ Foo): String = s"It a foo: $value" scala> onlyFoo("simple string") <console>:13: error: type mismatch; found : String("simple string") required: tag.@@[String,Foo] (which expands to) String with tag.Tagged[Foo] onlyFoo("simple string") ^ scala> val foo = tag[Foo]("Foo String") foo: tag.@@[String,Foo] = Foo String scala> onlyFoo(foo) res2: String = It a foo: Foo String def anyString(value: String): String = s"Just a string: $value” scala> anyString(foo) res6: String = Just a string: Foo String
  • 34.
    case class Agent(agentId:String @@ Agent, jobType: JobType, address: AgentAddress, status: AgentStatus, lastAccessed: Option[DateTime], inUse: Boolean, maybeInUseBy: Option[String @@ Job]) case class Job(referenceId: String @@ Job, jobType: JobType, status: JobStatus, submittedBy: User, submittedAt: DateTime, maybeStartedAt: Option[DateTime], maybeProcessedBy: Option[String @@ Agent], maybeCompletedAt: Option[DateTime])
  • 35.
    case class Job(referenceId:String @@ Agent, jobType: JobType, status: JobStatus, submittedBy: User, submittedAt: DateTime, maybeStartedAt: Option[DateTime], maybeProcessedBy: Option[String @@ Job], maybeCompletedAt: Option[DateTime])
  • 36.
    def recordCompletionMetrics(job: Job):Unit = { for( startedAt <- job.maybeStartedAt ; completedAt <- job.maybeCompletedAt ) { writeJobEvent( event = "Completed", time = completedAt, referenceId = job.referenceId, waitingTime = (startedAt - job.submittedAt), executionTime = (completedAt - startedAt)) } }
  • 37.
    def recordCompletionMetrics(job: Job):Unit = { require(job.status = JobComplete) require(job.maybeStartedAt.isDefined) require(job.maybeCompletedAt.isDefined) for (startedAt <- job.maybeStartedAt ; completedAt <- job.maybeCompletedAt ) { writeJobEvent ( event = "Completed", time = completedAt, referenceId = job.referenceId, waitingTime = (startedAt - job.submittedAt), executionTime = (completedAt - startedAt)) } }
  • 38.
    sealed trait Job{ def referenceId: String @@ Job def status: JobStatus def submittedBy: User def submittedAt: DateTime } case class WaitingJob(referenceId: String @@ Job, submittedBy: User, submittedAt: DateTime) extends Job { val status: JobStatus = JobWaiting }
  • 39.
    sealed trait Job{ def referenceId: String @@ Job def status: JobStatus def submittedBy: User def submittedAt: DateTime } case class ActiveJob(referenceId: String @@ Job, submittedBy: User, submittedAt: DateTime, startedAt: DateTime, processedBy: String @@ Agent) extends Job { val status: JobStatus = JobActive }
  • 40.
    sealed trait Job{ def referenceId: String @@ Job def status: JobStatus def submittedBy: User def submittedAt: DateTime } case class CompleteJob(referenceId: String @@ Job, submittedBy: User, submittedAt: DateTime, startedAt: DateTime, processedBy: String @@ Agent, completedAt: DateTime) extends Job { val status: JobStatus = JobComplete }
  • 41.
    def recordCompletionMetrics(job: CompleteJob):Unit = { writeJobEvent( event = "Completed", time = job.completedAt, referenceId = job.referenceId, waitingTime = (job.startedAt - job.submittedAt), executionTime = (job.completedAt - job.startedAt)) }
  • 42.
    Algebraic data types •Simply sealed traits and case classes • Exposes the shape of your data • Use this shape to control possible states
  • 43.
    More advanced techniques •Path dependent types • Self-recursive types • Phantom types • Shapeless
  • 44.
  • 45.
    “A mind islike a parachute, it doesn’t work unless its open” - Frank Zappa
  • 46.

Editor's Notes

  • #3 All my slides, notes and references are available on blog This take is about correctness
  • #4 Sounds complicate – Not – low hanging fruit – not functional – OOP In the 90s – Defensive programming Garbage in garbage out - Robust Tracing a bug back to its cause – Andy – Shawshank Redemption.
  • #5 Fail fast Find bugs quickly – Easier to fix Requires lots of tests Exceptions can reach customers
  • #6 Bertrand Meyer - Eiffle Design by Contract Systematic Fail fast - goal removes defensive programming tool support: property checking, JML, Spec#
  • #7 These techniques are about the code Correctness – choosing the correct data types – a good compile
  • #8 Maps – x Domain – y Range – example All of the domain – Total Function Code define domain + range with types – fucntion's signature – compiler Do not accept all possible values in their domain – fail fast, exceptions – return garbage – Domain swiss cheese Correctness is about choosing types that shrink the domain and the range Filling in the holes – functions "total” – Removing the chance of failure (construction / validation / compiler)
  • #9 I want to start by skipping through some simple rules for Correctness No detail Covered else where - links in blog
  • #10 Tony Hoare introduced Null references in ALGOL W back in 1965 “simply because it was so easy to implement”. He talks about that decision considering it “my billion-dollar mistake”. Simply use Option - Capture nulls from java code
  • #11 Nice properties – once constructed correctly it stays correct Actors
  • #12 Users don't care why it failed From Effective Java there are two types Business logic errors Capture these with types Option / Try / Validation / Or Programming Errors Prevent these with types Option / NonEmptyList / Type safe wrappe
  • #13 Wrapper types
  • #14 Currency is not a String – Only 176 international currencies (ISO 4217) They all have exactly three uppercase letters 17576 unique combinations of this rule, plain strings are effectively infinite
  • #15 Value class - Extends AnyVal Private constructor Currency.from returns an Option Enumeration
  • #16 Dangers of refactoring types – broad operations and type inference This function creditCardRate still compiles
  • #17 Can use ScalaZ or Scalactic triple equals Implementation
  • #18 Can use ScalaZ or Scalactic triple equals Implementation
  • #19 Rates and amounts have different semantics Amounts have a unit, like a currency, they can only be added and subtracted If you multiply two amounts you change the units A rate is a scalar, these values can be added, subtracted, multiplied and divided. Rates can multiply or divide amounts, scaling its size If you divide two amounts, you get a rate
  • #20 Now Rates cannot be added to amounts
  • #22 Primitively type code is weakly typed Validation – Reduces code and bugs
  • #23 … Lets look at a simple example
  • #25 Now the compiler prevents us from getting the average of an empty list
  • #26 Var-args are syntactic sugar for list arguments One plus var args == Non-empty var-args
  • #27 Cannot be empty – operation not supported – empty does not make sense Validation failures – banking customer’s list of accounts Compiler will tell everyone else
  • #28 Algebraic data types == traits and case classes
  • #29 Grid team – schedule and execute jobs on a cluster of agent machines Customer code in DMZ – Parallelize large jobs like payroll
  • #30 Here is a naive implementation of data model classes One to one mapping from database schema Options are prefixed with Maybe First lets remove some of the primitives
  • #33 Enforce with Tag types
  • #34 Tag types – shapeless – scalaz Leverage an optimization in scala compiler – the tag is removed at runtime Value remains the underlying type with extra information for type checking at compile time
  • #35 No control over construction or operations – Semantics require a wrapper type Best used for marking primary/foreign keys
  • #38 Fail fast – is not an option Must always record metrics
  • #42 Guaranteed to write metrics with the correct value Bugs where startedAt not set cause compile error
  • #43 If a function throws IllegalArgumentException or method throws IllegalStateException or InvalidOperationException Type is too broad consider breaking it up with an ADT
  • #44 A nested object’s type is specific to it parents instance – multi tenant – encryption Final type of a class is passed into a trait has a generic parameter – enums – generic repositories Any type that only exists at compile time – track objects state – enable/disable operations – type-safe builder A library that pushes the boundary of what’s possible at compile time
  • #45 Use types to constrain values and operations Compiler proves your code is correct – reduce tests – move validation to the correct place reduce the state space of your API making it easier to use Apply Defensive programming only where required Fail faster – at compile time DbC preconditions become types – but Immutability – don’t need invariants – post conditions become types also
  • #46 Scala is a very broad church – from OOP to FP Both have lots to teach – lots of techniques can be applied to both
  • #47 Slides, notes and references on the fledgling workday tech blog Any questions?