Introducing Zio DynamoDB - a new Scala library for Simple, type-safe, and efficient access to DynamoDB

Introducing zio-dynamodb
a new Scala library for DynamoDB
1 / 42

contributors
- Avinder Bahra
used Scala in production for last 5 years
used ZIO in production for the last 2 years
been programming professionally for last 30 years
- Adam Johnson
2 / 42

Some context
Relational Databases
pros
transactional (ACID), rich query language, tooling
3 / 42

Some context
Relational Databases
pros
transactional (ACID), rich query language, tooling
cons
not scalable
4 / 42

Some context
DynamoDB
"fully managed, serverless, key-value NoSQL database designed to
run high-performance applications at any scale"
5 / 42

Some context
DynamoDB
"fully managed, serverless, key-value NoSQL database designed to
run high-performance applications at any scale"
great for scalability, throughput with features like auto-scaling
and global table replication
6 / 42

Some context
DynamoDB
fully managed, serverless, key-value NoSQL database designed to run
high-performance applications at any scale
great for scalability, throughput with features like auto-scaling
and global table replication
downside of NoSQL DBs
they do less work than relational databases for tasks such as
querying and maintaining consistency/integrity and offer less
tooling
hence the code burden for the developer is significantly
increased
7 / 42

problem: using DDB Java SDK is painful
we are going to walk you through the process of creating a production
grade Scala application using the Java JDK.
Lets see how many LOC it takes to perform simple write and read
operations.
Here are the steps we are going to follow:
create a typesafe model (case class)
serialialse/deserialse the Scala model to the APIs model
create API requests for wring and reading the model to the database
execute the requests and process the results
improve performance by using batching of requests
as this is production grade we are going to use ZIO to manage effects,
concurrency, Java interop ...
8 / 42

final case class Student(email: String, // partition key
subject: String, // sort key
enrollmentDate: Option[Instant], payment: Payment)
sealed trait Payment
object Payment {
final case object DebitCard extends Payment
final case object CreditCard extends Payment
final case object PayPal extends Payment
}
9 / 42

we have to build a PutItemRequest
import scala.jdk.CollectionConverters._
final case class Student(email: String, subject: String,
object Payment {
}
def putItemRequest(student: Student): PutItemRequest =
PutItemRequest.builder
.tableName("student")
.item(toAttributeValueMap(student).asJava) //Item: Map[String, AttributeValue]
.build
10 / 42

we have to serialise the Student case class to AttributeValue's
final case class Student(email: String, subject: String,
object Payment {
}
def putItemRequest(student: Student): PutItemRequest =
PutItemRequest.builder
.item(toAttributeValueMap(student).asJava)
.build
def toAttributeValueMap(student: Student): Map[String, AttributeValue] = {
Map(
"email" -> AttributeValue.builder.s(student.email).build,
"subject" -> AttributeValue.builder.s(student.subject).build
)
}
11 / 42

we have to deal with optional and sum types
def toAttributeValueMap(student: Student): Map[String, AttributeValue] = {
val mandatoryFields = Map(
"subject" -> AttributeValue.builder.s(student.subject).build,
"payment" -> AttributeValue.builder.s { // serialise sum type
student.payment match {
case DebitCard => "DebitCard"
case CreditCard => "CreditCard"
case PayPal => "PayPal"
}
}.build
)
val nonEmptyOptionalFields: Map[String, AttributeValue] = Map(
"enrollmentDate" -> student.enrollmentDate.map(instant =>
AttributeValue.builder.s(instant.toString).build)
).filter(_._2.nonEmpty).view.mapValues(_.get).toMap
mandatoryFields ++ nonEmptyOptionalFields
}
12 / 42

Now that we have saved a Student as an Item in the DynamoDb database we
next need to create a GetItemRequest to retrieve it.
GetItemRequest.builder
.key(
Map(
"email" -> AttributeValue.builder.s("avi.gmail.com").build,
"subject" -> AttributeValue.builder.s("maths").build
).asJava
)
.build()
13 / 42

Next have to de-serialise a GetItemResponse - a Map[String, AttributeValue]
and deal with concerns such as:
mandatory fields - if not present we want an error
optional fields
conversion to standard primitive type eg Instant
Helper functions that return an Either[String, _] for managing errors
def getString(map: Map[String, AttributeValue],
name: String): Either[String, String] =
map.get(name).toRight(s"mandatory field $name not found").map(_.s)
def getStringOpt(map: Map[String, AttributeValue],
name: String): Either[Nothing, Option[String]] =
Right(map.get(name).map(_.s))
def parseInstant(s: String): Either[String, Instant] =
Try(Instant.parse(s)).toEither.left.map(_.getMessage)
14 / 42

we create a deserialise function that uses the previous helpers
note we have to de-serialise the Payment sum type
def deserialise(item: Map[String, AttributeValue]): Either[String, Student] =
for {
email <- getString(item, "email")
subject <- getString(item, "subject")
maybeEnrollmentDateAV <- getStringOpt(item, "enrollmentDate")
maybeEnrollmentDate <-
maybeEnrollmentDateAV.fold[Either[String, Option[Instant]]](Right(None))(s =>
parseInstant(s).map(i => Some(i))
)
payment <- getString(item, "payment")
paymentType = payment match {
case "DebitCard" => DebitCard
case "CreditCard" => CreditCard
case "PayPal" => PayPal
}
} yield Student(email, subject, maybeEnrollmentDate, paymentType)
15 / 42

stitching together all the functions using ZIO
val program =
for {
client <- ZIO.service[DynamoDbAsyncClient]
student = Student("avi@gmail.com", "maths", Instant.now,
Payment.DebitCard)
putRequest = putItemRequest(expectedStudent)
_ <- ZIO.fromCompletionStage(client.putItem(putRequest))
getItemRequest = getItemRequest(student)
getItemResponse <- ZIO.fromCompletionStage(client.getItem(getItemRequest))
studentItem = getItemResponse.item.asScala.toMap
foundStudent = deserialise(studentItemstudentItem)
} yield foundStudent
16 / 42

But that's not fast enough - we want to use DDB batching for our Puts and Gets
So we have to first create BatchWriteItemRequest
def batchWriteItemRequest(students: List[Student]): BatchWriteItemRequest = {
val putRequests = students.map { student =>
val request = PutRequest
.builder()
.item(toAttributeValueMap(student).asJava)
.build()
WriteRequest.builder().putRequest(request).build()
}
BatchWriteItemRequest
.builder()
.requestItems(Map("student" -> putRequests.asJava).asJava)
.build()
}
input is a List of Student which we map to WriteRequests
for each student se use our toAttributeValueMap function to serilaise
to an Item
finally we create a BatchWriteItemRequest
17 / 42

We then have to execute the batch and process the response
def batchWriteAndRetryUnprocessed(
batchRequest: BatchWriteItemRequest
): ZIO[Has[DynamoDbAsyncClient], Throwable, BatchWriteItemResponse] = {
val result = for {
response <- ZIO.fromCompletionStage(client.batchWriteItem(batchRequest))
} yield response
result.flatMap {
case response if response.unprocessedItems().isEmpty => ZIO.succeed(response)
case response =>
// very simple recursive retry of unprocessed requests
// in Production we would have exponential back offs and a timeout
batchWriteAndRetryUnprocessed(batchRequest =
BatchWriteItemRequest
.builder()
.requestItems(response.unprocessedItems())
.build()
)
}
}
18 / 42

We then have to create a BatchGetItemRequest
def batchGetItemReq(studentPks: Seq[(String, String)]): BatchGetItemRequest = {
val keysAndAttributes = KeysAndAttributes.builder
.keys(
studentPks.map { // for all students we extract the partition and sort keys
case (email, subject) =>
Map(
"email" -> AttributeValue.builder().s(email).build(),
"subject" -> AttributeValue.builder().s(subject).build()
).asJava
}.asJava
)
.build()
BatchGetItemRequest.builder
.requestItems(Map("student" -> keysAndAttributes).asJava)
.build()
}
19 / 42

We then have to execute the batch and process the response
def batchGetItemAndRetryUnprocessed(batchRequest: BatchGetItemRequest)
: ZIO[Has[DynamoDbAsyncClient], Throwable, BatchGetItemResponse] = {
val result = for {
response <- ZIO.fromCompletionStage(client.batchGetItem(batchRequest))
} yield response
result.flatMap {
case response if response.unprocessedKeys.isEmpty => ZIO.succeed(response)
case response =>
// very simple recursive retry of failed requests
// in Production we would have exponential back offs and a timeout
batchGetItemAndRetryUnprocessed(batchRequest =
BatchGetItemRequest.builder
.requestItems(response.unprocessedKeys)
.build
)
}
}
20 / 42

putting it all together - full batching program
val program =
for {
avi = Student("avi@gmail.com", ...)
adam = Student("adam@gmail.com", ...)
students = List(avi, adam)
batchPutRequest = batchWriteItemRequest(students)
_ <- batchWriteAndRetryUnprocessed(batchPutRequest)
batchGetItemResponse <-
batchGetItemAndRetryUnprocessed(batchGetItemRequest(
students.map(st => (st.email, st.subject))))
responseMap = batchGetItemResponse.responses.asScala
listOfErrorOrStudent =
responseMap.get("student")
.fold[List[Either[String, Student]]](List.empty) {
javaList =>
val listOfErrorOrStudent: List[Either[String, Student]] =
javaList.asScala.map(m =>
attributeValueMapToStudent(m.asScala.toMap)).toList
listOfErrorOrStudent
} // traverse!
errorOrStudents = foreach(listOfErrorOrStudent)(identity)
} yield errorOrStudents
21 / 42

That's not all, we want to speed up our updates
however there is no batching support in DDB for updates
So we want to parallelise our updates
def updateItemRequest(student: Student): UpdateItemRequest = {
val values: Map[String, AttributeValue] =
Map(":paymentType" ->
AttributeValue.builder.s(student.payment.toString).build)
UpdateItemRequest.builder
.key(
Map(
"subject" -> AttributeValue.builder.s(student.subject).build
).asJava
)
.updateExpression("set payment = :paymentType")
.expressionAttributeValues(values.asJava)
.build
}
...
// we execute two queries in parallel using a ZIO zipPar
ZIO.fromCompletionStage(client.updateItem(updateItemRequest(updatedAvi))) zipPar
ZIO.fromCompletionStage(client.updateItem(updateItemRequest(updatedAdam))
)
22 / 42

Thats a lot of boilerplate!
...and this is just the tip of the iceberg - we have not covered
// TODO: should I create more examples for any of these?
// TODO: show how many lines of code
// TODO: move though slides quickly @ explain at a higher level
scanning and queries with key condition and filter expressions
pagination of scan and query results
complex projection expressions
error handling
TODO: solution is
23 / 42

solution: zio-dynamodb
Simple, type-safe, and efficient access to DynamoDB
We are now going to write the equivelent application using zio-dynamodb and
show you how much less boilerplate code there is.
24 / 42

val program = (for {
avi = Student("avi@gmail.com", "maths", ...)
adam = Student("adam@gmail.com", "english", ...)
_ <- (DynamoDBQuery.put("student", avi) zip
DynamoDBQuery.put("student", adam)).execute
listErrorOrStudent <- DynamoDBQuery
.forEach(List(avi, adam)) { st =>
DynamoDBQuery.get[Student](
"student",
PrimaryKey("email" -> st.email, "subject" -> st.subject)
)
}
.execute
} yield EitherUtil.collectAll(listErrorOrStudent))
.provideCustomLayer(DynamoDBExecutor.live)
25 / 42

offers a type safe API with auto serialisation
auto batching and parallelisation queries
testable using a fake in memory DB
26 / 42

zio-dynamodb API 101
DynamoDBQuery
its type and it's combinators
auto batching and parallelisation
how to create and execute a query
Serialisation
low level API
built in type classes
Expressions
usages
mutations operations
query operations
Serialisation
high level type safe API
note there is a 1:1 correspondence to the AWS API to aid discoverability
27 / 42

DynamoDBQuery
sealed trait DynamoDBQuery[+A] {
def zip[B](that: DynamoDBQuery[B]): DynamoDBQuery[(A, B)] = ??
def zipLeft[B](that: DynamoDBQuery[B]): DynamoDBQuery[A] = ???
def zipRight[B](that: DynamoDBQuery[B]): DynamoDBQuery[B] = ???
def forEach[A, B](values: Iterable[A])(body: A => DynamoDBQuery[B])
: DynamoDBQuery[List[B]]
def execute: ZIO[Has[DynamoDBExecutor], Exception, A] = ???
}
object DynamoDBQuery {
// whole bunch of query constructors
}
Next lets create and execute some basic queries
28 / 42

Simple Put for multiple tables
val program = (for {
_ <- (DynamoDBQuery.put("student", avi) zip
DynamoDBQuery.put("student", adam) zip
DynamoDBQuery.put("course", french) zip
DynamoDBQuery.put("course", art) zip
).execute
} yield ())
.provideCustomLayer(DynamoDBExecutor.live)
puts for multiple tables are batch together, grouped by table in the request
29 / 42

Serialisation - Low level API
AttributeValue
sealed trait AttributeValue
final case class Binary(value: Iterable[Byte]) extends AttributeValue
final case class BinarySet(value: Iterable[Iterable[Byte]]) extends AttributeValue
final case class Bool(value: Boolean) extends AttributeValue
// ... etc etc
final case class String(value: ScalaString) extends AttributeValue
final case class Number(value: BigDecimal) extends AttributeValue
This corresponds 1:1 with the AWS API AttributeValue
30 / 42

Serialisation - Low level API
AttrMap
Top level container type for an Item with type aliases
final case class AttrMap(map: Map[String, AttributeValue])
type Item = AttrMap
type PrimaryKey = AttrMap
Internal type classes take care of AttributeValue conversions
val aviItem = Item("email" -> "avi@gmail.com", "age" -> 21)
is equivalent to
val aviPrimaryKey = AttrMap(Map("email" -> AttributeValue.String("email"),
"age" -> AttributeValue.Number(BigDecimal(21)))
31 / 42

Projection Expression Parser
$("cost") // simple
$("address.line1") // map
$("adresses[1]") // list
$("adresses[work].line1") // list with map
The $ projection expression parser function takes a string field expression and
turns it into the internal representation
32 / 42

updates
updateItem("course", PrimaryKey("name" -> "art"))(
// UpdateExpression
$("cost").set(500.0) + $("code").set("123")
)
we can specify ProjectionExpression's
$("cost") ... $("code")
a ProjectionExpression has may update actions which can be combined using
+
$("count").set(1)
$("field1").set($("field2")) // replaces filed1 with field2
$("count").setIfNotExists($("two"), 42)
$("numberList").appendList(List("1"))
$("numberList").prependList(List("1"))
$("count").add(1) // updating Numbers and Sets
$("count").remove // Removes this field from an item
$("numberSet").deleteFromSet(1)
$("person.address").set(Item("line1" -> "1 high street"))
33 / 42

delete
deleteItem("course", PrimaryKey("name" -> "art"))
Both Update and Delete can have ConditionExpressions that must be met for
the operation to succeed. For this we use the where method.
where $("code") > 1 && $("code") < 5
applied to the queries
deleteItem("course", PrimaryKey("name" -> "art")) where
$("code") > 1 && $("code") < 5
updateItem("course", PrimaryKey("name" -> "art")) {
// UpdateExpression
$("cost").set(500.0) + $("code").set("123")
} where $("code") > 1 && $("code") < 5
34 / 42

queryAll
val zio: ZIO[Has[DynamoDBExecutor], Exception, Stream[Exception, Person]] =
queryAll("person", $("name"), $("address[1].line1"))
.whereKey(
PartitionKey("partitionKey1") === "x" && SortKey("sortKey1") > 10
)
.execute
Note that we use a list of ProjectionExpression's again
$("name"), $("address[1].line1")
The whereKey method specifies a KeyConditionExpression
.whereKey(
PartitionKey("partitionKey1") === "x" && SortKey("sortKey1") > 10
)
... and we get back a ZStream that the library lazily paginates for us
val queryAll: ZIO[Has[DynamoDBExecutor], Exception, Stream[Exception, Person]]
35 / 42

querySome
val zio =
querySomeItem("person", limit = 5, $("name"), $("address[1].line1"))
.whereKey(PartitionKey("partitionKey1") === "x" && SortKey("sortKey1") > 10)
.execute
... and we get back a ZIO of (Chunk[Person], LastEvaluatedKey)
type LastEvaluatedKey = Option[PrimaryKey]
val q: ZIO[Has[DynamoDBExecutor], Exception, (Chunk[Item], LastEvaluatedKey)]
and we use the startKey method to feed back the LastEvaluatedKey to get the
next page of data
val zio =
querySomeItem("person", limit = 5, last$("name"), $("address[1].line1"))
.whereKey(PartitionKey("partitionKey1") === "x" && SortKey("sortKey1") > 10)
.startKey(startKey)
.execute
36 / 42

scanAll, scanSome
scanAll
These are similar to the previous queryAll/querySome queries
val zio: ZIO[Has[DynamoDBExecutor], Exception, Stream[Exception, Person]] =
scanAll("person", $("name"), $("address[1].line1")).execute
scanSome
val zio =
scanSome("person", limit = 5, $("name"), $("address[1].line1")).execute
...
val zio =
scanSome("person", limit = 5, $("name"), $("address[1].line1"))
.startKey(startKey)
.execute
37 / 42

Condition Expressions
$("field1").exists
$("field1").notExists
$("field1").beginsWith("1")
$("field1").contains("1")
$("field1").size > 1
$("field1").isNumber
condition1 && condition2
condition1 || condition2
!conition1
$("field1") > 1.0
$("field1") > $("col2")
$("field1") === $("col2")
$("field1") === "2"
$("field1").between("1", "2")
$("field1").in(Set("1", "2"))
$("field1").in("1", "2")
38 / 42

Condition Expressions cont.
Apply to
updateItem
delete
queryXXXX/scanXXXX
39 / 42

ConditionExpressions for query and scan
40 / 42

Serialisation - High level API
High level API uses zio-schema to create codecs
41 / 42

summary (3 slide)
wrap-up (1 slide)
Java SDK is painful
ZIO DynamoDB is a joy
Learning more (1 slide)
Thank you (1 slide)
42 / 42

Introducing Zio DynamoDB - a new Scala library for Simple, type-safe, and efficient access to DynamoDB

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Introducing Zio DynamoDB - a new Scala library for Simple, type-safe, and efficient access to DynamoDB

Similar to Introducing Zio DynamoDB - a new Scala library for Simple, type-safe, and efficient access to DynamoDB (20)

Recently uploaded

Recently uploaded (20)

Introducing Zio DynamoDB - a new Scala library for Simple, type-safe, and efficient access to DynamoDB