Introducing the ZIO DynamoDB Type Safe API

Introducing the type-safe API for ZIO
DynamoDB
2 / 68

- Avinder Bahra
https://github.com/googley42
3 / 68

Talk Outline
quick recap on using the AWS Java SDK
where we got up to in last years presentation - what was missing?
quick tour of the low level API
introduction to the new Type-safe API
show examples using the new API
summary and wrap up
4 / 68

Using AWS Java SDK recap
5 / 68

AWS SDK example
final case class Student(
email: String, // partition key
enrollmentDate: Option[Instant],
payment: Payment
)
sealed trait Payment
object Payment {
final case object DebitCard extends Payment
final case object CreditCard extends Payment
final case object PayPal extends Payment
}
creates a Student DynamoDB table
serialises above model to the DynamoDB taking into account
optional fields
sum types
integrates with AWS Java SDK and does Java interop
does batched writes and reads
...what does the code look like?...
6 / 68

Lots of bolierplate code!
page 1 ... ... page 4!!!
240 lines of code!
7 / 68

AWS SDK example
Sumary
- lots of tedious boilerplate code
- not type safe
- prone to run time errors
8 / 68

ZIO DynamoDB solution
val liveDynamoDBExecutorLayer = ...
val errorOrListOfStudent = (for {
avi = Student("avi@gmail.com", "maths", ...)
adam = Student("adam@gmail.com", "english", ...)
_ <- (put("student", avi) zip put("student", adam)).execute
listOfErrorOrStudent <- forEach(List(avi, adam)) { st =>
get[Student]("student",
PrimaryKey("email" -> st.email, "subject" -> st.subject)
)}.execute
} yield EitherUtil.collectAll(listOfErrorOrStudent))
.provideLayer(liveDynamoDBExecutorLayer)
10 / 68

Tour of Low Level API
AttributeValue's
ProjectionExpressions's
ConditionExpressions's
UpdateExpressions's
11 / 68

AttributeValue
An AttributeValue is DynamoDB type and value pair that is stored in the
database
sealed trait AttributeValue
final case class Binary(value: Iterable[Byte]) extends AttributeValue
final case class BinarySet(value: Iterable[Iterable[Byte]]) extends AttributeValue
final case class Bool(value: Boolean) extends AttributeValue
// ... etc etc
final case class String(value: ScalaString) extends AttributeValue
final case class Number(value: BigDecimal) extends AttributeValue
12 / 68

Item (AttrMap)
A DynamoDB Item is record - a map of field name to AttributeValue. In the
low level API it is modelled as an AttrMap
final case class AttrMap(map: Map[String, AttributeValue])
type Item = AttrMap
type PrimaryKey = AttrMap
val aviItem1 = AttrMap(Map("email" -> AttributeValue.String("email"),
"age" -> AttributeValue.Number(BigDecimal(21)))
// using an apply method that takes a type class
val aviItem2 = AttrMap("email" -> "avi@gmail.com", "age" -> 21)
val bool = aviItem1 == aviItem2 // true
13 / 68

DynamoDBQuery trait
sealed trait DynamoDBQuery[+A] {
def execute: ZIO[Has[DynamoDBExecutor], Exception, A] = ???
def zip[B](that: DynamoDBQuery[B]): DynamoDBQuery[(A, B)] = ???
...
def forEach[A, B](values: Iterable[A])(body: A => DynamoDBQuery[B])
: DynamoDBQuery[List[B]]
}
trait DynamoDBExecutor {
def execute[A](query: DynamoDBQuery[A]): ZIO[Any, Throwable, A]
}
final case class GetItem(...) extends DynamoDBQuery[Option[Item]]
// etc etc
14 / 68

ProjectionExpression
In DynamoDB a projection expression is a way of accessing the attribute
email
address.line1
tutors[1].surname
In the API it is represented as a sealed trait
sealed trait ProjectionExpression {...}
final case class Root extends ProjectionExpression
final case class MapElement(...) extends ProjectionExpression
final case class ListElement(...) extends ProjectionExpression
15 / 68

$ function (type unsafe)
manually creating ProjectionExpression's using the constructors is painful
ListElement(MapElement(MapElement(Root("foo"), "bar"), "baz"), 9) === 42
16 / 68

so we introduced the $ function
$("foo.bar.baz") === 42 && $("students[0].firstname") === "Avi"
17 / 68

so we introduced the $ function
$("foo.bar.baz") === 42 && $("students[0].firstname") === "Avi"
which is really convenient but totally type unsafe
18 / 68

Condition and Filter Expressions
In DynamoDB a condition expression is a way specifying a condition that must
hold for a PutItem, UpdateItem, or DeleteItem
A very similar concept is a filter expression that applies to a DynamoDB Query
In ZIO DynamoDB both Condition and Filter expressions concepts are modelled
by:
sealed trait ConditionExpression{...}
19 / 68

Condition and Filter Expressions Syntax
20 / 68

Update Expressions
update-expression ::=
[ SET action [, action] ... ]
[ REMOVE action [, action] ...]
[ ADD action [, action] ... ]
[ DELETE action [, action] ...]
set-action ::=
path = value
value ::=
operand
| operand '+' operand
| operand '-' operand
operand ::= path | function
function
list_append (list1, list2)
if_not_exists (path, value)
remove-action ::= path
add-action ::= path value
delete-action ::= path value
21 / 68

Example of Low Level API
val avi = AttrMap("email" -> "avi@a.com", "payment" -> "DebitCard", "age" -> 42)
val adam = AttrMap("email" -> "adam@a.com", "payment" -> "DebitCard", "age" -> 21)
22 / 68

private val program = for {
_ <- (putItem("student", avi) zip putItem("student", adam)).execute
} yield ()
23 / 68

item <- getItem("student", PrimaryKey("email" -> "avi@a.com")).execute
} yield ()
24 / 68

_ <- updateItem("student", PrimaryKey("email" -> "avi@a.com")){
$("payment").set("Paypal") + $("age").add(1)
}.execute
} yield ()
25 / 68

_ <- updateItem("student", PrimaryKey("email" -> "avi@a.com")) {
}
_ <- deleteItem("student", PrimaryKey("email" -> "avi@a.com")).where(
$("payment") === "Paypal" && $("age") > 21
).execute
} yield ()
26 / 68

Overview of Low Level APIs - Example
item <- getItem("student", PrimaryKey("email" -> "a@b.com")).execute
_ <- updateItem("student", PrimaryKey("email" -> "a@b.com")){
}
_ <- deleteItem("student", PrimaryKey("email" -> "a@b.com")).where(
$("payment") === "Paypal" && $("age") > 21
).execute
student: Student = tediousCustomDeSerialisationCode(item)
} yield student
27 / 68

Type Safe Serialisation API using ZIO Schema
ZIO Schema
"Compositional, type-safe schema definitions, which enable auto-
derivation of codecs and migrations."
28 / 68

final case class Student(email: String, subject: String, age: Int)
object Student {
implicit lazy val schema: Schema[Student] = DeriveSchema.gen[Student]
}
val avi = Student("avi@gmail.com", "maths", 21)
val adam = Student("adam@gmail.com", "english", 21)
29 / 68

object Student {
implicit lazy val schema: Schema[Student] = DeriveSchema.gen[Student]
}
val avi = Student("avi@gmail.com", "maths", 21)
val adam = Student("adam@gmail.com", "english", 21)
private val program = (for {
_ <- (put("student", avi) zip put("student", adam)).execute
student <- get("student", primaryKey("avi@gmail.com", "maths"))
.execute.right
} yield student == avi /* true */ )
type classes to derive codecs for serialisation to DynamoDB for put and
get
note that we are serialising case class directly
this is a huge increase in usability!
30 / 68

Codec customisations using annotations
annotations for cutomising
tagged union vs discrimated union
enum storage
names
case class
case class fields
discriminator field
enum values
31 / 68

Where we left off in the last talk
val avi: Student = get("student", primaryKey)
put[Student]("student", Student("avi@gmail.com", ...),...)
.where( $("addr.line1").contains("street") && $("addr.line2").beginsWith("f") )
updateItem("student", primaryKey) {
$("firstname").set("Avinder") +
$("groups").appendList(Chunk("group1"))
}.where( $("email") === "avi@gmail.com" )
deleteItem("student", primaryKey)
.where( $("age") > "18" && $("email") === "avi@gmail.com") )
querySomeItem("tableName1", limit = 10, $("email"), $("payment"), $("age"))
.sortOrder(ascending = false)
.whereKey(PartitionKey("email") === "avi@gmail.com")
.filter( $("age") > 21 && $("subject").beginsWith("Math") )
Can you spot the run time error problem in the above code?
32 / 68

Where we left off in the last talk
34 / 68

Introducing the new type safe API
35 / 68

Optics
Provide a way of navigating your immutable data structure that reduces
boilerplate
Prism - access product data
Lens - access sum type data
Traversal - access a collection
Drilling down into a field eg Student.email is a Scala language feature that is
not a value
Optics libraries take these language features and turn them into values that
can be composed
Typically libraries use use functions in the encoding (executable encoding)
36 / 68

Reified Optics
sealed trait Payment
object Payment {
final case object DebitCard extends Payment
// ...
implicit val schema = DeriveSchema.gen[Payment]
}
enrollDate: Instant,
payment: Payment
)
object Student {
implicit val schema = DeriveSchema.gen[Student]
val (email, enrollDate, payment) = ProjectionExpression.accessors[Student]
}
37 / 68

Reified Optics
Reified optics are not enough on their own - we needed extend this type safety
to the rest of the API
so we added phantom type parameters to our lower level representations:
sealed trait ProjectionExpression[-From,
+To]
sealed trait ConditionExpression[-From]
38 / 68

New type safe API
the new type safe API uses reified optics to constrain expressions to the
type we are working with
39 / 68

New type safe API - Update Expressions
val student = Student(...)
object Student {
val (email, enrollDate, payment, age) = ProjectionExpression.accessors[Student]
}
The update expressions here are typesafe.
40 / 68

New type safe API - Condition Expressions
object Student {
}
41 / 68

New type safe API - delete condition expressions
object Student {
}
Elephant injection attack will not work!
42 / 68

New Type-Safe API examples
43 / 68

Example 1 - Optimistic locking
44 / 68

Optimistic locking example
final case class Student(email: String, payment: Payment, age: Int,
version: Long = 0)
45 / 68

version: Long = 0)
object Student {
val (email, payment, age, version) = ProjectionExpression.accessors[Student]
}
46 / 68

version: Long = 0)
object Student {
}
def optimisticUpdate(primaryKey: PrimaryKey)(f: Student => Action[Student]) = ???
47 / 68

version: Long = 0)
object Student {
}
def optimisticUpdate(primaryKey: PrimaryKey)(f: Student => Action[Student]) =
for {
before <- get[Student]("student", primaryKey).execute.absolve
} yield ()
48 / 68

version: Long = 0)
object Student {
}
for {
_ <- update[Student]("student", primaryKey) { ... }.execute
} yield ()
49 / 68

version: Long = 0)
object Student {
}
for {
_ <- update[Student]("student", primaryKey) {
f(before) + version.add(1)
} ...
} yield ()
50 / 68

version: Long = 0)
object Student {
}
for {
}.where(Student.version === before.version).execute.retry(policy)
} yield ()
51 / 68

version: Long = 0)
object Student {
}
for {
}.where(Student.version === before.version).execute.retry(policy)
} yield ()
for {
_ <- optimisticUpdate(primaryKey("avi@gmail.com", "maths")) { student =>
if (student.age > 42)
Student.payment.set(Payment.DebitCard)
else
Student.payment.set(Payment.CreditCard)
}.execute
} yield ()
52 / 68

Example 2 - Migration Pipeline Example
53 / 68

Migration Pipeline Example
age: Int,
enrollmentDate: Instant,
payment: Payment
)
final case class StudentCourses(
email: String,
courses: Set[String]
)
final case class Student2(
age: Int,
payment: Payment,
)
54 / 68

age: Int,
payment: Payment
)
email: String,
)
age: Int,
payment: Payment,
)
55 / 68

age: Int,
payment: Payment
)
email: String,
)
age: Int,
payment: Payment,
)
56 / 68

students <- scanAll[Student](tableName = "s")
.parallel(16)
.filter(Student.age > 21 && Student.enrollmentDate > aDate)
.execute
studentsAndCourses =
batchReadFromStream[...]("sc", students)(s => PrimaryKey("email" -> s.email))
_ <- batchWriteFromStream(studentsAndCourses) {
case (s, sc) => put(tableName = "Students2",
Student2(s.email, s.age, s.enrollmentDate, s.payment, sc.courses)
)
}.runDrain
} yield ()
57 / 68

.parallel(16)
.execute
studentsAndCourses =
)
}.runDrain
} yield ()
58 / 68

.parallel(16)
.execute
studentsAndCourses: ZStream[..., (Student, StudentCourses)] =
)
}.runDrain
} yield ()
59 / 68

.parallel(16)
.execute
studentsAndCourses: ZStream[..., (Student, StudentCourses)] =
)
}.runDrain
} yield ()
60 / 68

Summary - Condition and Filter Expression
62 / 68

Summary - Type Safety coverage
63 / 68

Summary - Update Expressions
set-action ::=
path = value
value ::=
operand
function
64 / 68

Summary - Type Safety coverage
set-action ::=
path = value
value ::=
operand
function
65 / 68

Summary - New Type Safe API
66 / 68

Learning More
github repo https://github.com/zio/zio-dynamodb
microsite https://zio.github.io/zio-dynamodb/
examples in own sbt module zio-dynamodb/examples
integration tests dynamodb/src/it/scala/zio/dynamodb/
last years talk introducing ZIO DynamoDB at Functional Scala 2021
67 / 68

Thanks
a big thank you to:
all the sponsors Ziverge, Scalac, UMATR, Coralogix, Conducktor,
Matechs for giving me this opportunity to talk about this library
Sandra and Agata for organising the conference
John for his mentorship during development
68 / 68

Introducing the ZIO DynamoDB Type Safe API

Recommended

Recommended

More Related Content

Similar to Introducing the ZIO DynamoDB Type Safe API

Similar to Introducing the ZIO DynamoDB Type Safe API (20)

Recently uploaded

Recently uploaded (20)

Introducing the ZIO DynamoDB Type Safe API