Intudunction to three essential functional programming techniques that our team applied in our recent projects.

1. Functional programming
techniques in real-world
microservices
07. 04. 2017

2. Andras Papp
Tech Lead @

3. THE B2C
MARKETING CLOUD
truly personalised
interactions
250,000+ campaigns / month
10+ billion messages /month
2+ billion end users

4. The tale of Prince Scalarot

5. The abandoned castle
of Princess Monada

6. The three trials

7. How should I change specific
values of a complex immutable
data structure?
Question 1

10. Unified profile of a customer
BigQuery
Unified Profile
Service
Products
Campaign names
…
Events

12. ContactProfile(
lastEvents = LastEvents(
click = LastEvent(campaign = Campaign(id = 100), "iPhone")
…
),
purchases = PurchaseInfo(
last = Some(SinglePurchase(
campaign = Campaign(id = 101), quantity = 2
))
…
)
…
…
)

13. ContactProfile(
lastEvents = LastEvents(
click = LastEvent(campaign = Campaign(id = 100), “iPhone")
…
),
purchases = PurchaseInfo(
last = Some(SinglePurchase(
campaign = Campaign(id = 100), quantity = 2
))
…
)
…
…
)

14. Transformation
Campaign(id = 100)
Campaign(
id = 100,
name = "Abandoned Shopping Cart Campaign”
)

15. Immutable domain entities
Always creating a copy:
val campaignWithName = campaign.copy(
name = "Abandoned Shopping Cart Campaign”
)

16. Copyception
… and we have only updated ONE campaign
contactProfile.copy(
lastEvents = contactProfile.lastEvents.copy(
send = contactProfile.lastEvents.send.copy(
campaign = replaceCampaignWithName(campaignNames)
(contactProfile.lastEvents.send.campaign)
)
)
)

17. Lens
Type of the object to update is O
Type of the field to update is V
case class Lens[O, V] (
get: O => V,
set: (O, V) => O
)

18. Example lens instance
val campaignNameLens = Lens[Campaign, String](
get = _.name,
set = (campaign, name) => campaign.copy(name = name)
)
val campaign = Campaign(id = 1, name = "old name")
campaignNameLens.set(campaign, "new name")
// Campaign(1, "new name")

19. Composing lenses
def compose[Outer, Inner, Value](
outer: Lens[Outer, Inner],
inner: Lens[Inner, Value]
) = Lens[Outer, Value](
get = outer.get andThen inner.get,
set = (obj, value) => outer.set(
obj,
inner.set(outer.get(obj), value)
)
)

20. Composing lenses
val lastSendEvent = LastEvent(
campaign = Campaign(id = 100, name = “"),
device = “iPhone"
)
val lastEventCampaignLens = Lens[LastEvent, Campaign](
get = _.campaign,
set = (lastEvent, campaign) =>
lastEvent.copy(campaign = campaign)
)
val campaignNameLens = Lens[Campaign, String](
get = _.name,
set = (campaign, name) =>
campaign.copy(name = name)
)
val lastEventCampaignNameLens = compose(
lastEventCampaignLens, campaignNameLens)
lastEventCampaignNameLens.set(lastSendEvent, "Campaign Name")

21. Lens implementations
Monocle
Quicklens
Shapeless
Scalaz

22. Solution of Sir Lensalot
Using composed lenses

23. ContactProfile(
lastEvents = LastEvents(
click = LastEvent(campaign = Campaign(id = 100), “iPhone")
…
),
purchases = PurchaseInfo(
last = Some(SinglePurchase(
campaign = Campaign(id = 100), quantity = 2
))
…
)
…
…
)

24. val clickCampaignLens =
lens[ContactProfile]
>> 'lastEvents
>> 'click
>> ‘campaign
val lastPurchaseLens =
lens[ContactProfile]
>> 'purchases
>> ‘last
Solving the problem
Define lenses using shapeless

25. def modifyBy(lens: Lens[ContactProfile, Campaign]) =
(contactProfile: ContactProfile) =>
lens.modify(contactProfile)
(replaceCampaignWithName(campaignNames))
val contactProfileWithCampaignNames =
(
modifyBy(clickCampaignLens) andThen
modifyBy(lastPurchaseLens)
)(contactProfile)
Solving the problem

27. How can I convert between a third
party library’s representation and
my case classes in a generic way?
Question 2

29. def executeQuery(query: Query): Future[List[TableRow]] =
{
queryResult(query).map { result =>
if (result.getTotalRows.equals(BigInteger.ZERO)) {
List.empty[TableRow]
} else {
result.getRows.asScala.toList
}
}
}
Google BigQuery API
Running a query job in BigQuery results
in a list of TableRow objects

30. case class ClickEvent(campaignId: Int, eventTime: DateTime,
deviceName: String, customerId: Int, messageId: Int)
def loadClicks(contact: Contact): Future[List[ClickEvent]] = {
executeQuery(createQuery).map(_.map(
tableRow => {
val cells = tableRow.getF
ClickEvent(
campaignId = cells.get(0).getV.toString.toInt
eventTime = new DateTime(cells.get(1).getV.toString.toLong),
deviceName = cells.get(1).getV.toString
)
}
))
}
TableRow - model
We would like to work with our models

31. Solution of Sir General
Using generic programming

32. def executeQuery[T](query: Query): Future[List[T]] = {
queryResult(query).map { result =>
if (result.getTotalRows.equals(BigInteger.ZERO)) {
List.empty[T]
} else {
result.getRows.asScala.toList.map(_.as[T])
}
}
}
def loadClicks(contact: Contact): Future[List[ClickEvent]] =
executeQuery[ClickEvent](createQuery(contact, clickTable))
Aim for a generic solution

33. Step 1: Move mapping
to type classes

34. trait BigQueryFormat[T] {
def fromTableRow(tableRow: TableRow): T
}
implicit val clickEventFormat = new BigQueryFormat[ClickEvent] {
def fromTableRow(tableRow: TableRow) = {
val cells = tableRow.getF
ClickEvent(
campaignId = cells.get(0).getV.toString.toInt,
eventTime = new DateTime(cells.get(1).getV.toString.toLong),
deviceName = cells.get(2).getV.toString
)
}
}
BigQueryFormat type class
BigQueryFormat trait and one simplified
concrete instance for ClickEvent

35. object syntax {
implicit class RichTableRow(val row: TableRow)
extends AnyVal {
def as[T]
(implicit format: BigQueryFormat[T]): T =
format.fromTableRow(row)
}
}
Syntax
Add ‘as’ function to TableRow

36. "create ClickEvent case class from TableRow" in {
val tableRow = new TableRow()
val now = DateTime.now
tableRow.setF(List[TableCell](
new TableCell().setV(1),
new TableCell().setV(now.getMillis),
new TableCell().setV("iPhone")
).asJava)
tableRow.as[ClickEvent] shouldEqual
ClickEvent(1, now, "iPhone")
}
Test it!
Convert a test TableRow to a case class
with our syntax: ’as’

37. What have we
achieved?

38. Step 2: Get rid of all
the mapping code

39. Sir General
visited the
sorcerer

40. The plan

41. Type class for cell values
trait BigQueryValue[T] {
def fromValue(v: AnyRef): T
}
implicit object stringPrimitive extends BigQueryValue[String] {
def fromValue(v: AnyRef) = v.toString
}
implicit object intPrimitive extends BigQueryValue[Int] {
def fromValue(v: AnyRef) = v.toString.toInt
}
implicit object BoolPrimitive extends BigQueryValue[Boolean] {
def fromValue(v: AnyRef) = v.toString.toBoolean
}
implicit object DatePrimitive extends BigQueryValue[DateTime] {
def fromValue(v: AnyRef) = new DateTime(v.toString.toLong)
}
…

42. BigQueryFormat for HList
General representation
ClickEvent(1, now, "iPhone")
1 :: now :: "iPhone" :: HNil

43. implicit object hNilBigQueryFormat
extends BigQueryFormat[HNil] {
def fromTableRow(m: TableRow) = HNil
}
BigQueryFormat for HList
Convert to the members of the HList
recursively, ultimately to HNil

44. implicit def hListBigQueryFormat[Head, Tail <: HList](
implicit valueFormatter: BigQueryValue[Head],
tailFormatter: BigQueryFormat[Tail]
): BigQueryFormat[Head :: Tail] = {
new BigQueryFormat[Head :: Tail] {
def fromTableRow(tableRow: TableRow) = {
val tableCells = tableRow.getF.toList
val cellValue = tableCells.head.getV
val resolvedValue = valueFormatter.fromValue(cellValue)
tableRow.setF(tableCells.tail.asJava)
val tail = tailFormatter.fromTableRow(tableRow)
resolvedValue :: tail
}
}
}
BigQueryFormat for HList
But what about HList?

45. implicit def hListBigQueryFormat[Head, Tail <: HList](
implicit valueFormatter: BigQueryValue[Head],
tailFormatter: BigQueryFormat[Tail]
): BigQueryFormat[Head :: Tail] = {
new BigQueryFormat[Head :: Tail] {
def fromTableRow(tableRow: TableRow) = {
val tableCells = tableRow.getF.toList
val cellValue = tableCells.head.getV
val resolvedValue = valueFormatter.fromValue(cellValue)
tableRow.setF(tableCells.tail.asJava)
val tail = tailFormatter.fromTableRow(tableRow)
resolvedValue :: tail
}
}
}
BigQueryFormat for HList

46. implicit def hListBigQueryFormat[Head, Tail <: HList](
implicit valueFormatter: BigQueryValue[Head],
tailFormatter: BigQueryFormat[Tail]
): BigQueryFormat[Head :: Tail] = {
new BigQueryFormat[Head :: Tail] {
def fromTableRow(tableRow: TableRow) = {
val tableCells = tableRow.getF.toList
val cellValue = tableCells.head.getV
val resolvedValue = valueFormatter.fromValue(cellValue)
tableRow.setF(tableCells.tail.asJava)
val tail = tailFormatter.fromTableRow(tableRow)
resolvedValue :: tail
}
}
}
BigQueryFormat for HList
What is happening?

47. Generic BigQueryFormat
Generic type class instance
BigQueryFormat for generic
representation
Create HList representation
Convert to expected type

48. Generic BigQueryFormat
implicit def genericBigQueryFormat[T, Repr](
implicit gen: Generic.Aux[T, Repr],
reprFormatter: BigQueryFormat[Repr]
): BigQueryFormat[T] = {
new BigQueryFormat[T] {
def fromTableRow(tableRow: TableRow) = {
val repr = reprFormatter.fromTableRow(tableRow)
gen.from(repr)
}
}
}
What is Aux?

49. Nice job!
We can remove all the concrete type
class instances for our model classes!

50. Related topics
Support ADTs
Field names of case classes
Handle conversion errors

51. Our library is open source!
https://github.com/emartech/
googlecloud-shapeless
Contribution is welcome!

53. How can I write pure functions
if I have to do a bunch of IO
operations?
Question 3

55. Overview
Microservices do a lot of API calls
There is a data store involved in most
cases
An IO operation depends on the result
of another IO operation

56. A sample side-
effecting function

57. A sample side-effecting
function
def create(customerId: Int,
fieldsToUse: List[ContactField],
contactApiClient: ContactFieldApi
): Future[List[FieldItem]] = {
for {
existingFields <- contactApiClient.list(customerId).map(_.data)
fieldRequests = FieldPayload.create(existingFields, fieldsToUse)
newFields <- createNewFields(fieldRequests, contactApiClient)
} yield newFields
}
private def createNewFields(
fieldRequests: List[CreateFieldRequest],
contactApiClient: ContactFieldApi
): Future[List[FieldItem]] = {
Future.sequence(fieldRequests.map(request => {
contactApiClient
.createField(request)
.map(createFieldItem(request))
}))
}

58. A sample side-effecting
function
def create(customerId: Int,
fieldsToUse: List[ContactField],
contactApiClient: ContactFieldApi
): Future[List[FieldItem]] = {
for {
existingFields <- contactApiClient.list(customerId).map(_.data)
fieldRequests = FieldPayload.create(existingFields, fieldsToUse)
newFields <- createNewFields(fieldRequests, contactApiClient)
} yield newFields
}
private def createNewFields(
fieldRequests: List[CreateFieldRequest],
contactApiClient: ContactFieldApi
): Future[List[FieldItem]] = {
Future.sequence(fieldRequests.map(request => {
contactApiClient
.createField(request)
.map(createFieldItem(request))
}))
}

59. Issues with this approach
Hard to unit test
Dependency management
complications
Business logic & the environment
Future on the interface
Future in function body

60. Solution of Sir Effectus
Using type classes

61. Type classes again!
Let’s create a type class for type classes!

62. RestIO type class
trait RestIO[F[_]] {
def listFields(customerId: Int): F[List[FieldItem]]
def createField(payload: CreateFieldRequest): F[CreateFieldResponse]
}
object RestIO {
object syntax {
def listFields[F[_]](customerId: Int)(implicit ev: RestIO[F]) =
ev.listFields(customerId)
def createField[F[_]](payload: CreateFieldRequest)
(implicit ev: RestIO[F]) =
ev.createField(payload)
}
}

63. Instance for Future
object ClientInstance {
implicit val suiteClientInstance = new RestIO[Future]
{
val contactApiClient = ContactFieldApi()
def listFields(customerId: Int)
: Future[List[FieldItem]] =
contactApiClient.list(customerId).map(_.data)
def createField(payload: CreateFieldRequest)
: Future[CreateFieldResponse] =
contactApiClient.createField(customerId, payload)
}
}

64. Changes in client code
[error] value map is not a member of type parameter F[List[FieldItem]]
import RestIO.syntax._
def create[F[_]: RestIO](
customerId: Int,
fieldsToUse: List[ContactField]
): F[List[FieldItem]] = {
for {
existingFields <- listFields(customerId)
fieldRequests = FieldPayload.create(existingFields, fieldsToUse)
newFields <- createNewFields(fieldRequests)
} yield newFields
}
The interface is generic for the type class
Client class dependency is removed

65. Does it work?
[error] value map is not a member of type parameter F[List[FieldItem]]
import RestIO.syntax._
def create[F[_]: RestIO](
customerId: Int,
fieldsToUse: List[ContactField]
): F[List[FieldItem]] = {
for {
existingFields <- listFields(customerId)
fieldRequests = FieldPayload.create(existingFields, fieldsToUse)
newFields <- createNewFields(fieldRequests)
} yield newFields
}

66. Monad to the rescue!
[error] value map is not a member of type parameter F[List[FieldItem]]
import cats._
import RestIO.syntax._
def create[F[_]: RestIO: Monad](
customerId: Int,
fieldsToUse: List[ContactField]
): F[List[FieldItem]] = {
for {
existingFields <- listFields(customerId)
fieldRequests =
FieldPayload.create(existingFields, fieldsToUse)
newFields <- createNewFields(fieldRequests)
} yield newFields
}
Let’s use cats!

67. Changes in client code
[error] value map is not a member of type parameter F[List[FieldItem]]
private def createNewFields[F[_]: RestIO: Monad](
customerId: Int,
fieldRequests: List[CreateFieldRequest]
): F[List[FieldItem]] = {
fieldRequests.map(request => {
createField(request)
.map(createFieldItem(request))
}).sequence
}
Got rid of Future.sequence

68. Testing
new instance of RestIO for testing
simple stub
Id monad

69. Let’s create an instance for
testing!
trait FakeRestIO {
val expectedFieldListResponse
: Map[Int, List[FieldItem]] = Map()
…
implicit val fakeClientInstance = new RestIO[Id] {
override def createField(payload: CreateFieldRequest)
: Id[CreateFieldResponse] = {
expectedCreateFieldResponse(payload)
}
…
}
}

70. A sample test case
"create segment with new custom field" in new FakeRestIO {
override val expectedFieldListResponse = Map(
customerId -> List(
FieldItem(101, Some("Custom Email"))
)
)
val fieldsToUse = List(
ContactField("Custom Field"),
ContactField("Custom Email")
)
override val expectedCreateFieldResponse = Map(
CreateFieldRequest("Custom Field") ->
CreateFieldResponse(1001)
)
val expectedNewField = List(
FieldItem(1001, Some("Custom Field”))
)
create(customerId, fieldsToUse) shouldBe expectedNewField
}

71. Conclusions
More generic
Business logic has no side effects
Testing without Future
Dependency management by the
compiler

73. Learn more about type
classes
https://github.com/hablapps/
typeclassesforthemasses
Hablapps

74. Learn more about shapeless
https://github.com/underscoreio/
shapeless-guide

75. Thank You!
Q&A