Kraków Scala User Group Meetup

January 10, 2019
Sync considered
unethical
Word about me
• Tomasz Kogut
• Tech lead at research
• I work on internal machine learning platform
• Scala since 2011
• almendar @
Kraków Scala User Group Meetup

January 10, 2019
Sync considered
unethical
Kraków Scala User Group Meetup

January 10, 2019
Sync considered
unethical
…or something like that
Cats Sync
Side effects suspension
• Sync[F] - Monix Task, ZIO, Cats-IO
• Future, Try
val f1 = Future(Random.nextInt)
val r1 = map2(f1,f1)(_+_)
val r11 = map2(
Future(Random.nextInt),
Future(Random.nextInt)
)(_+_)
val f2 = IO(Random.nextInt)
val r2 = map2(f2,f2)(_+_)
val r22 = map2(
IO(Random.nextInt),
IO(Random.nextInt)
)(_+_)
Equational reasoning
val f1 = Future(Random.nextInt) //2
val r1 = map2(f1,f1)(_+_) //4
val r11 = map2(
Future(Random.nextInt), //2
Future(Random.nextInt) //3
)(_+_) //5
//4 != 5
val f2 = IO(Random.nextInt) //() => Int
val r2 = map2(f2,f2)(_+_) //() => Int
val r22 = map2(
IO(Random.nextInt), //() => Int
IO(Random.nextInt) //() => Int
)(_+_) //() => Int
//r2.unsafeRunSync != r22.unsafeRunSync
Equational reasoning
def store[F[_]: Sync](folder: Path): F[CommErr[Path]] =
for {
_ <- Sync[F].delay(folder.toFile.mkdirs())
fileName <- Sync[F].delay(UUID.randomUUID.toString + "_packet.bts")
file = folder.resolve(fileName)
fos <- Sync[F].delay(new FileOutputStream(file.toFile))
orErr <- Sync[F].delay {
fos.write(packet.toByteArray)
fos.flush()
}.attempt
resErr <- orErr match {
case Left(th) =>
gracefullyClose(fos) *> Left(unableToStorePacket(packet, th)).pure[F]
case Right(_) =>
gracefullyClose(fos) map {
case Left(th) => Left(unableToStorePacket(packet, th))
case Right(_) => Right(file)
}
}
} yield resErr
Sample code
def store(folder:Path): CommErr[Path] = {
var fos: FileOutputStream = null
try {
folder.toFile.mkdirs()
val fileName = UUID.randomUUID.toString + "_packet.bts"
val file = folder.resolve(fileName)
val fos = new FileOutputStream(file.toFile)
fos.write(packet.toByteArray)
fos.flush()
gracefullyClose(fos) map {
case Left(th) => Left(unableToStorePacket(packet, th))
case Right(_) => Right(file)
}
} catch {
case ex:Exception =>
if(fos != null) gracefullyClose(fos)
Left(unableToStorePacket(packet, th))
}
}
Sample code
Rewrite
• John Backus, 1977 Turing Award lecture, “Can Programming
Be Liberated from the von Neumann Style? A functional
style and its algebra of programs”
• Conventional programming languages are growing ever more
enormous, but not stronger. Inherent defects at the most
basic level cause them to be both fat and weak: their
primitive word-at-a-time style of programming inherited
from their common ancestor–the von Neumann computer,
their close coupling of semantics to state transitions,
their division of programming into a world of expressions
and a world of statements, their inability to effectively
use powerful combining forms for building new programs
from existing ones, and their lack of useful mathematical
properties for reasoning about programs.
Von Neumann
alive and still kicking
Von Neumann
• John Backus, 1977 Turing Award lecture, “Can Programming
Be Liberated from the von Neumann Style? A functional
style and its algebra of programs”
• Conventional programming languages are growing ever more
enormous, but not stronger. Inherent defects at the most
basic level cause them to be both fat and weak: their
primitive word-at-a-time style of programming inherited
from their common ancestor–the von Neumann computer,
their close coupling of semantics to state transitions,
their division of programming into a world of expressions
and a world of statements, their inability to effectively
use powerful combining forms for building new programs
from existing ones, and their lack of useful mathematical
properties for reasoning about programs.
alive and still kicking
Hardware underneath
IO Monad
Functional Programming
• No order of execution
• No notion of time
• Ordered independent evaluation
• I/O - stateful and sequential
Is monadic IO the answer?
• Conal Elliott, http: //conal.net/blog/
• Many publication in Haskell since 1995
• e.g.Functional Reactive Programming
Is monadic IO the answer?
Is monadic IO the answer?
type IO a = RealWorld -> (a, RealWorld)
main :: RealWorld -> ((), RealWorld)
main :: RealWorld -> ((), RealWorld)
main world0 = let (a, world1) = getChar world0
(b, world2) = getChar world1
in ((), world2)
Is monadic IO the answer?
Is monadic IO the answer?
Speaking of IO
• Did you know Haskell has exceptions?
• Throwing exceptions is pure (!!!), function
only depends on body + args
• Caching them is not, so can be done only in
IO
Exceptions
Speaking of IO
forkIO :: IO () -> IO ThreadId
throwTo :: Exception e => ThreadId -> e -> IO ()
timeout :: Int -> IO a -> IO (Maybe a)
def forkIO(io:IO[Unit]): IO[ThreadId]
def throwTo(threadId: ThreadId, Exception e): IO[Unit]
def timeout[A](microseconds:Int, ioa: IO[A]): IO[Option[A]]
Async Exceptions
-- Executes an IO computation with asynchronous exceptions masked.
-- That is, any thread which attempts to raise an exception in the
-- current thread with throwTo will be blocked until asynchronous
-- exceptions are unmasked again.
mask :: ((forall a. IO a -> IO a) -> IO b) -> IO b
Masking exceptions
• Monad Transformers
• MTL
• Free Monads
• Eff
• Tagless final
• ???
FP Techniques
Monad Transformers
trait Env
trait AuditEvent
type AuditLog = Vector[AuditEvent]
trait Errors
type ValidationError = NonEmptyList[Errors]
type Metrics = Int
type Capability[A] = ReaderT[IO, Env, A]
type Audited[A] = WriterT[Capability, AuditLog, A]
type Measured[A] = StateT[Audited, Metrics, A]
type Action[A] = EitherT[Measured, ValidationError, A]
def log(ae: AuditEvent): Action[Unit] = EitherT[ValidationError](
StateT.set(
WriterT.put(
ReaderT.ask((env:Env) =>
IO.pure({}))(
Vector(ae)))))
Monad Transformers
Scala 

Slow, lot of allocations, hard to use, unreadable
Haskell

WriterT

space leaks
StateT 

exceptions

thread-safety 

put 4 >> concurrently (modify (+ 1)) (modify (+ 2)) >> get
ReaderT - used widely
Other: ResourceT, ExceptT, ClientT
MTL
trait Env
trait AuditEvent
type AuditLog = Vector[AuditEvent]
trait Errors
type ValidationError = NonEmptyList[Errors]
type Metrics = Int
type Capability[A] = ReaderT[IO, Env, A]
type Audited[A] = WriterT[Capability, AuditLog, A]
type Measured[A] = StateT[Audited, Metrics, A]
type Action[A] = EitherT[Measured, ValidationError, A]
def log[F[_]: MonadState : MonadReader : MonadWriter: MonadError]: F[Unit] {
val ms = MonadState[F]
val md = MonadReader[F]
val mw = MonadWriter[F]
val me = MonadError[F]
for {
( ...)
} yield
}
MTL
• IORef / Ref are becoming a backbone
• MonadWriter

PureScript WriterT -> IORef 10x mem reduction
• MonadReader

Useful for classy optics
• MonadState

IORef solves StateT problems
sealed abstract class DockerOp[A] extends Product with Serializable
object DockerOp {
final case class Extract(unit: UnitDef) extends DockerOp[Image]
final case class Tag(image: Image, registry: RegistryURI) extends DockerOp[(Int, Image)]
final case class Push(image: Image) extends DockerOp[(Int, List[Docker.Push.Output])]
final case class Pull(image: Image) extends DockerOp[(Int, List[Docker.Pull.Output])]
type DockerF[A] = Free[DockerOp, A]
}
class Docker(cfg: DockerConfig,
scheduler: ScheduledExecutorService,
ec: ExecutionContext)
extends (DockerOp ~> IO) {
def apply[A](op: DockerOp[A]): IO[A] =
op match {
case DockerOp.Tag(i, r) => tag(i, i.to(r)).retryExponentially()(scheduler, ec)
case DockerOp.Push(i) => push(i).retryExponentially()(scheduler, ec)
case DockerOp.Pull(i) => pull(i).retryExponentially()(scheduler, ec)
case DockerOp.Extract(unit) => extract(unit)
}
Free Monads
Nelson
final class KubernetesShell(
kubectl: Kubectl,
timeout: FiniteDuration,
executionContext: ExecutionContext,
scheduledES: ScheduledExecutorService
) extends (SchedulerOp ~> IO)
final class NomadHttp(
cfg: NomadConfig,
nomad: Infrastructure.Nomad,
client: org.http4s.client.Client[IO],
scheduler: ScheduledExecutorService,
ec: ExecutionContext
) extends (SchedulerOp ~> IO)
sealed abstract class SchedulerOp[A]
type SchedulerF[A] = Free[SchedulerOp, A]
final case class Delete(dc: Datacenter, d: Datacenter.Deployment) extends
SchedulerOp[Unit]
final case class Launch(i: Image, dc: Datacenter,
ns: NamespaceName, a: UnitDef @@ Versioned,
p: Plan, hash: String) extends SchedulerOp[String]
final case class Summary(dc: Datacenter, ns: NamespaceName,
sn: Datacenter.StackName) extends SchedulerOp[Option[DeploymentSummary]]
Free Monads
Nelson
trait FileAlg[F[_]] {
def deleteForce(file: File): F[Unit]
def ensureExists(dir: File): F[File]
def home: F[File]
def removeTemporarily[A](file: File)(fa: F[A]): F[A]
def readFile(file: File): F[Option[String]]
def walk(dir: File): Stream[F, File]
def writeFile(file: File, content: String): F[Unit]
def writeFileData(dir: File, fileData: FileData): F[Unit] =
writeFile(dir / fileData.name, fileData.content)
}
trait SbtAlg[F[_]] {
def addGlobalPlugin(plugin: FileData): F[Unit]
def addGlobalPlugins: F[Unit]
def getDependencies(repo: Repo): F[List[Dependency]]
def getUpdatesForProject(project: ArtificialProject): F[List[Update.Single]]
def getUpdatesForRepo(repo: Repo): F[List[Update.Single]]
}
Tagless-final
Scala Steward
final case class Context[F[_]](
config: Config,
dependencyService: DependencyService[F],
fileAlg: FileAlg[F],
filterAlg: FilterAlg[F],
gitAlg: GitAlg[F],
gitHubApiAlg: GitHubApiAlg[F],
logger: Logger[F],
nurtureAlg: NurtureAlg[F],
processAlg: ProcessAlg[F],
sbtAlg: SbtAlg[F],
updateService: UpdateService[F],
workspaceAlg: WorkspaceAlg[F]
)
Tagless-final
Scala Steward
def run(args: List[String]): IO[ExitCode] =
Context.create[IO](args).use { ctx =>
ctx.logger.infoTotalTime("run") {
for {
repos <- readRepos[IO](ctx.config.reposFile)
_ <- prepareEnv(ctx)
_ <- repos.traverse(ctx.dependencyService.forkAndCheckDependencies)
allUpdates <- ctx.updateService.checkForUpdates(repos)
reposToNurture <- ctx.updateService.

filterByApplicableUpdates(repos, allUpdates)
_ <- IO(reposToNurture.map(_.show).foreach(println))
_ <- IO(println(reposToNurture.size))
_ <- reposToNurture.
filter(repos.contains).traverse_(ctx.nurtureAlg.nurture)
//_ <- repos.traverse_(ctx.nurtureAlg.nurture)
} yield ExitCode.Success
}
Tagless-final
Scala Steward
Domain Specific Languages
trait StorageOp[A]
object StorageOp {
type StorageF[A] = Free[StorageOp, A]
final case class Get(key:String)
extends StorageOp[Option[String]]
final case class Put(key:String, data:String)
extends StorageOp[Unit]
final case class Remove(key:String)
extends StorageOp[Unit]
def get(key: String): StorageF[Option[String]] =
Free.liftF(Get(key))
def put(key: String, data: String): StorageF[Unit] =
Free.liftF(Put(key, data))
def remove(key: String): StorageF[Unit] =
Free.liftF(Remove(key))
}
trait Storage[F[_]] {
def get(key: String): F[Option[String]]
def put(key: String, data: String): F[Unit]
def remove(key: String): F[Unit]
}
def TFProgram[F[_]: Monad : Storage]: F[Unit] = {
val storage: Storage[F] = implicitly
import storage._
for {
_ <- put("key1", "data1")
data <- get("key1")
_ <- remove("key1")
} yield ()
}
def FreeProgram: StorageF[Unit] = {
import StorageOp._
for {
_ <- put("key1", "data1")
data <- get("key1")
_ <- remove("key1")
} yield ()
}
Domain Specific Languages
def freeCompiler: StorageOp ~> IO =
new (StorageOp ~> IO) {
val mock = new MockStorage
import mock._
def apply[A](fa: StorageOp[A]): IO[A] =
fa match {
case Put(key, value) => put(key, value)
case Get(key) => get(key)
case Remove(key) => remove(key)
}
}
def TFCompiler: Storage[IO] = new Storage[IO] {
val mock = new MockStorage
import mock._
def get(key: String): IO[Option[String]] = get(key)
def put(key: String, value: String): IO[Unit] = put(key, value)
def remove(key: String): IO[Unit] = remove(key)
}
}
class MockStorage {
private val kvs = mutable.Map.empty[String, String]
def get(key: String): IO[Option[String]] = {
IO.pure(kvs.get(key))
}
def put(key: String, value: String): IO[Unit] = {
kvs(key) = value
IO.unit
}
def remove(key: String): IO[Unit]= {
kvs.remove(key)
IO.unit
}
}
def store[F[_]: Sync](folder: Path): F[CommErr[Path]] =
for {
_ <- Sync[F].delay(folder.toFile.mkdirs())
fileName <- Sync[F].delay(UUID.randomUUID.toString + "_packet.bts")
file = folder.resolve(fileName)
fos <- Sync[F].delay(new FileOutputStream(file.toFile))
orErr <- Sync[F].delay {
fos.write(packet.toByteArray)
fos.flush()
}.attempt
resErr <- orErr match {
case Left(th) =>
gracefullyClose(fos) *> Left(unableToStorePacket(packet, th)).pure[F]
case Right(_) =>
gracefullyClose(fos) map {
case Left(th) => Left(unableToStorePacket(packet, th))
case Right(_) => Right(file)
}
}
} yield resErr
Sample code
Three Layer App
Layer 1
• The Real-World layer
• No business logic
• Infra where created application run on
• Logging, Metrics, Http client, Database, S3
HTTP
abstract class MonadHttpClient[F[_]] {
def get(url: Url): F[Data]
def post[A: ToJson](url: Url, a: A): F[Data]
}
HTTP
abstract class MonadHttpClient[F[_]] {
def get(url: Url): F[Data]
def post[A: ToJson](url: Url, a: A): F[Data]
}
HTTP
class ProdHttpClient extends MonadHttpClient[IO] {
def get(url: Url): IO[Data] =
def post[A: ToJson](url: Url, a: A): IO[Data] =
}
Production
HTTP
abstract class MockHttp[A] { self =>
(…)
}
Testing
HTTP
abstract class MockHttp[A] { self =>
def map[B](f: A => B): MockHttp[B] = ???
def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ???
}
HTTP
abstract class MockHttp[A](env: ???) { self =>
def map[B](f: A => B): MockHttp[B] = ???
def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ???
}
HTTP
abstract class MockHttp[A](env: HttpEnv) { self =>
def map[B](f: A => B): MockHttp[B] = ???
def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ???
}
type HttpEnv = Id[HttpState]
type HttpState = ConcurrentHashMap[Url, Data]
HTTP
abstract class MockHttp[A] { self =>
def runMockHttp: ReaderT[IO, HttpEnv, A] = ???
def map[B](f: A => B): MockHttp[B] = ???
def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ???
}
type HttpEnv = Id[HttpState]
type HttpState = ConcurrentHashMap[Url, Data]
HTTP
abstract class MockHttp[A] { self =>
def runMockHttp: ReaderT[IO, HttpEnv, A]
def map[B](f: A => B): MockHttp[B] = new MockHttp[B] {
override def runMockHttp: ReaderT[IO, HttpEnv, B] =
self.runMockHttp.map(f)
}
def flatMap[B](f: A => MockHttp[B]): MockHttp[B] =
new MockHttp[B] {
override def runMockHttp: ReaderT[IO, HttpEnv, B] =
self.runMockHttp.flatMap(a => f(a).runMockHttp)
}
}
type HttpEnv = Id[HttpState]
type HttpState = ConcurrentHashMap[Url, Data]
HTTP
implicit def monadInstance: Monad[MockHttp] = ???
implicit def instance: MonadHttpClient[MockHttp] = ???
HTTP
implicit def monadInstance: Monad[MockHttp] = new Monad[MockHttp] {
override def flatMap[A, B](fa: MockHttp[A])(f: A => MockHttp[B]): MockHttp[B] = fa.flatMap(f)
override def pure[A](x: A): MockHttp[A] = new MockHttp[A] {
override def runMockHttp: ReaderT[IO, HttpEnv, A] = ReaderT[IO, HttpEnv, A](_ => IO.pure(x))
}
}
implicit def instance: MonadHttpClient[MockHttp] = ???
HTTP
implicit def monadInstance: Monad[MockHttp] = new Monad[MockHttp] {
override def flatMap[A, B](fa: MockHttp[A])(f: A => MockHttp[B]): MockHttp[B] = fa.flatMap(f)
override def pure[A](x: A): MockHttp[A] = new MockHttp[A] {
override def runMockHttp: ReaderT[IO, HttpEnv, A] = ReaderT[IO, HttpEnv, A](_ => IO.pure(x))
}
}
implicit def instance: MonadHttpClient[MockHttp] = new MonadHttpClient[MockHttp] {
override def get(url: Url): MockHttp[Data] = new MockHttp[Data] {
override def runMockHttp: ReaderT[IO, HttpEnv, Data] = ReaderT { env: HttpEnv =>
IO(env.getOrDefault(url, "404 NotFound"))
}
}
override def post[A: ToJson](url: Url, a: A): MockHttp[Data] = new MockHttp[Data] {
override def runMockHttp: ReaderT[IO, HttpEnv, Data] = ReaderT { env: HttpEnv =>
IO(env.put(url, implicitly[ToJson[A]].encode(a)))
}
}
}
Why MockHttp
• Limiting yourself
• Creating DSL
• Not possible to use Sync[F] !!!
Why MockHttp
• Limiting yourself
• Creating DSL
• Not possible to use Sync[F] !!!
Logging
abstract class MonadLogging[F[_]] {
def info(s: String): F[Unit]
def warning(s: String): F[Unit]
}
final case class LoggingConfig(context: Class[_])
abstract class JavaLogging[A] { self =>
def run: ReaderT[IO, LoggingConfig, A]
def map[B](f: A => B): JavaLogging[B] = new JavaLogging[B] {
override def run: ReaderT[IO, LoggingConfig, B] = self.run.map(f)
}
def flatMap[B](f: A => JavaLogging[B]): JavaLogging[B] = new JavaLogging[B] {
override def run: ReaderT[IO, LoggingConfig, B] = self.run.flatMap(a => f(a).run)
}
}
implicit val monadInstance = new Monad[JavaLogging] {
override def pure[A](x: A): JavaLogging[A] = new JavaLogging[A] {
override def run: ReaderT[IO, LoggingConfig, A] = ReaderT[IO, LoggingConfig, A](_ => IO.pure(x))
}
override def flatMap[A, B](fa: JavaLogging[A])(f: A => JavaLogging[B]): JavaLogging[B] = fa.flatMap(f)
}
implicit def instance: MonadLogging[JavaLogging] = new MonadLogging[JavaLogging] {
override def info(s: String): JavaLogging[Unit] = new JavaLogging[Unit] {
override def run: ReaderT[IO, LoggingConfig, Unit] =
ReaderT(x => IO(Logger.getLogger(x.context.getName).info(s)))
}
override def warning(s: String): JavaLogging[Unit] = new JavaLogging[Unit] {
override def run: ReaderT[IO, LoggingConfig, Unit] =
ReaderT(x => IO(Logger.getLogger(x.context.getName).warning(s)))
}
}
Running
def mockHttpRequest [F[_]: MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ???
def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ???
Running
def mockHttpRequest [F[_]: MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ???
def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ???
run :: forall f a. MonadHttpClient f => f a -> IO a
def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A]
Running
def mockHttpRequest [F[_]: MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ???
def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ???
run :: forall f a. MonadHttpClient f => f a -> IO a
def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A]
Rank-N types
Running
def mockHttpRequest [F[_]: MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ???
def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ???
run :: forall f a. MonadHttpClient f => f a -> IO a
def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A]
Rank-N types
run :: forall a. (forall f. MonadHttp f => f a) -> IO a
Running
def mockHttpRequest [F[_]: MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ???
def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ???
run :: forall f a. MonadHttpClient f => f a -> IO a
def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A]
Rank-N types
run :: forall a. (forall f. MonadHttp f => f a) -> IO a
def run[A](ma:F[A] forAll {type F[_]}): IO[A]
HttpService
abstract class HttpService[F[_]] {
def runHttp[A](fa:F[A]): IO[A]
}
HttpService
abstract class HttpService {
type F[_]
def runHttp[A](fa:F[A]): IO[A]
}
HttpService
abstract class HttpService {
type HttpEff[_]
def runHttp[A](fa:HttpEff[A]): IO[A]
}
HttpService
abstract class HttpService {
type HttpEff[_]
val MH: MonadHttpClient[HttpEff]
def runHttp[A](fa:HttpEff[A]): IO[A]
}
HttpService
abstract class HttpService {
type HttpEff[_]
val MH: MonadHttpClient[HttpEff]
implicit val HttpEffM: Monad[HttpEff]
def runHttp[A](fa:HttpEff[A]): IO[A]
}
Services
abstract class Services {
val http: HttpService
val logging: LoggingService
val metrics: MetricsServic
val s3Storage: StorageService
}
def mockServices(): Services = new Services {
val httpEnv: HttpEnv = new ConcurrentHashMap()
override val http: HttpService = new HttpService {
override type HttpEff[A] = MockHttp[A]
override implicit val a: Monad[HttpEff] = MockHttp.monadInstance
override implicit val MH: MonadHttpClient[HttpEff] = MockHttp.instance
override def runHttp[A](fa:MockHttp[A]): IO[A] = fa.runMockHttp.run(httpEnv)
(…)
}
Application
case class Application[A](unwrap: ReaderT[IO, Services, A]) {
def run(services: Services): IO[A] = unwrap.run(services)
}
object Application {
def apply[A](f: Services => IO[A]): Application[A] =
Application(ReaderT[IO, Services, A](f))
implicit def monadInstance: Monad[Application] = new Monad[Application] {
override def flatMap[A, B](fa: Application[A])
(f: A => Application[B]): Application[B] =
new Application(fa.unwrap.flatMap { (a: A) =>
f(a).unwrap
})
override def pure[A](x: A): Application[A] = Application(_ => IO.pure(x))
}
}
def logWarning(msg: String, ctx: Class[_] = this.getClass) =
Application[Unit] {
services: Services =>
import services.logging._
runLogging[Unit](ctx)(
for {
_ <- ML.warning(msg)
} yield ()
)
}
def downloadS3KeyFromWebService: Application[Data] =
Application { services: Services =>
import services.http._
runHttp {
for {
_ <- MH.post("http: //xyz.com",
Random.alphanumeric.take(Random.nextInt(15)).mkString)
i <- MH.get("http: //xyz.com")
} yield i
}
}
object Main extends IOApp {
val program: Application[ExitCode] = for {
key <- downloadS3KeyFromWebService
_ <- logWarning(s"Key downloaded: ${key}")
_ <- bumpCounterMetrics(IncCounter("Http", key.length))
m1 <- readCounter(ReadValue("Http"))
_ <- logWarning(s"Number of bytes downloaded $m1")
dh <- readFromS3(s"s3: //rpp/test/bucket/$key")
bytesSaved <- storeOnLocalDisk("output.txt", dh)
_ <- logWarning(s"Number of bytes saved $bytesSaved")
} yield ExitCode.Success
override def run(args: List[String]): IO[ExitCode] = {
program.run(Services.mockServices())
}
}
Interpretation
abstract class HttpService {
type HttpEff[_]
def runHttp[A](fa:HttpEff[A]): IO[A]
}
Interpretation
abstract class HttpService {
type HttpEff[_]
def runHttp[A]: InterpreterFor[IO, HttpEff]
}
type InterpreterFor[F[_], G[_]] = FunctionK[G, F]
Interpretation
abstract class HttpService {
type HttpEff[_]
def runHttp[A]: IO InterpreterFor HttpEff
}
type InterpreterFor[F[_], G[_]] = FunctionK[G, F]
Interpretation
abstract class HttpService {
type HttpEff[_]
def runHttp[A]: IO InterpreterFor HttpEff
}
type InterpreterFor[F[_], G[_]] = FunctionK[G, F]
abstract class Services {
val http: IO InterpreterFor HttpEff
val logging: IO InterpreterFor LogEff
val metrics: IO InterpreterFor MetricEff
val s3Storage: IO InterpreterFor StorageEff
}
Interpretation
abstract class HttpService {
type HttpEff[_]
def runHttp[A]: IO InterpreterFor HttpEff
}
type InterpreterFor[F[_], G[_]] = FunctionK[G, F]
abstract class Services[G[_]:Sync] {
val http: G InterpreterFor HttpEff
val logging: G InterpreterFor LogEff
val metrics: G InterpreterFor MetricEff
val s3Storage: G InterpreterFor StorageEff
}
Interpretation
abstract class HttpService {
type HttpEff[_]
def runHttp[A]: IO InterpreterFor HttpEff
}
type InterpreterFor[F[_], G[_]] = FunctionK[G, F]
abstract class Services {
val http: Free[HttpF, ?] InterpreterFor HttpEff
val logging: Free[LoggingF, ?] InterpreterFor LogEff
val metrics: Free[MetricsF, ?] InterpreterFor MetricEff
val s3Storage: Free[StorageF, ?] InterpreterFor StorageEff
}
Layer 2
• Business Domain Resources
• Things that work with external world but
for some purpose
• Examples: UserRepository, Payment processor
apis
Layer 2 example
trait AbExperiments[F[_]] {
def getVersion(algoArea: String, tag: String): F[Version]
}
Layer 2 example
trait AbExperiments[F[_]] {
def getVersion(algoArea: String, tag: String): F[Version]
}
trait AlgoTrainer[F[_]] {
def train(version: Version): F[Unit]
}
Layer 2 example
trait AbExperiments[F[_]] {
def getVersion(algoArea: String, tag: String): F[Version]
}
trait AlgoTrainer[F[_]] {
def train(version: Version): F[Unit]
}
override def run(args: List[String]): IO[ExitCode] = {
def trainProgram[F[_]: Monad](implicit algoTrainer: AlgoTrainer[F],
abExperiments: AbExperiments[F]): F[Unit] = for {
version <- abExperiments.getVersion("algoTag", "tag")
_ <- algoTrainer.train(version)
} yield ()
val trainProgramApp = trainProgram[Application].map(_ => ExitCode.Success)
trainProgramApp.run(Services.mockServices())
}
Layer 2 example
implicit def abExperiments: AbExperiments[Application] =
(algoArea: String, tag: String) =>
Application { services =>
import services.http._
import services.logging._
for {
_ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training"))
v <- runHttp(MH.get(s"$algoArea/$tag"))
} yield v
}
implicit def algoTrainApp: AlgoTrainer[Application] =
(version: Version) =>
Application { services =>
import services.logging._
runLogging(AlgoTrainer.getClass)(ML.info(s"Training $version"))
}
Layer 2 example
type AppT[F[_], Ctx, A] = ReaderT[F, Ctx, A]
type Getter[S, A] = S => A
type Has[A, Ctx] = Getter[Ctx,A]
implicit def abExperiments: AbExperiments[Application] =
(algoArea: String, tag: String) =>
Application { services =>
import services.http._
import services.logging._
for {
_ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training"))
v <- runHttp(MH.get(s"$algoArea/$tag"))
} yield v
}
Layer 2 example
type AppT[F[_], Ctx, A] = ReaderT[F, Ctx, A]
type Getter[S, A] = S => A
type Has[A, Ctx] = Getter[Ctx,A]
implicit def abExperiments: AbExperiments[Application] =
(algoArea: String, tag: String) =>
Application { services =>
import services.http._
import services.logging._
for {
_ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training"))
v <- runHttp(MH.get(s"$algoArea/$tag"))
} yield v
}
implicit def abExperiments[F[_]: Monad, Ctx](implicit
httpG: Has[HttpService[F], Ctx],
loggingG: Has[LoggingService[F], Ctx]): AbExperiments[AppT[F, Ctx, ?]] =
(algoArea: String, tag: String) =>
ReaderT[F, Ctx, Version] { (ctx: Ctx) =>
val http: HttpService[F] = httpG(ctx)
val logging: LoggingService[F] = loggingG(ctx)
import http._
import logging._
for {
_ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training"))
v <- runHttp(MH.get(s"$algoArea/$tag"))
} yield v
}
Layer 3
• Business logic only
• All pure functions
• All data already obtained
• No problem to use Option, Either and
friends
• Probably not type class based
Monad Effects
Ethics
• No hard evidence that types work
• Might be immutability
• Clean room process
• COCOMO: Upfront design reduces 10k defect
by 10%
• Coq
• TLA+
Thank you!
Questions?
almendar @
Tomasz Kogut

Sync considered unethical

  • 1.
    Kraków Scala UserGroup Meetup
 January 10, 2019 Sync considered unethical
  • 2.
    Word about me •Tomasz Kogut • Tech lead at research • I work on internal machine learning platform • Scala since 2011 • almendar @
  • 3.
    Kraków Scala UserGroup Meetup
 January 10, 2019 Sync considered unethical
  • 4.
    Kraków Scala UserGroup Meetup
 January 10, 2019 Sync considered unethical …or something like that
  • 7.
  • 8.
    Side effects suspension •Sync[F] - Monix Task, ZIO, Cats-IO • Future, Try
  • 9.
    val f1 =Future(Random.nextInt) val r1 = map2(f1,f1)(_+_) val r11 = map2( Future(Random.nextInt), Future(Random.nextInt) )(_+_) val f2 = IO(Random.nextInt) val r2 = map2(f2,f2)(_+_) val r22 = map2( IO(Random.nextInt), IO(Random.nextInt) )(_+_) Equational reasoning
  • 10.
    val f1 =Future(Random.nextInt) //2 val r1 = map2(f1,f1)(_+_) //4 val r11 = map2( Future(Random.nextInt), //2 Future(Random.nextInt) //3 )(_+_) //5 //4 != 5 val f2 = IO(Random.nextInt) //() => Int val r2 = map2(f2,f2)(_+_) //() => Int val r22 = map2( IO(Random.nextInt), //() => Int IO(Random.nextInt) //() => Int )(_+_) //() => Int //r2.unsafeRunSync != r22.unsafeRunSync Equational reasoning
  • 11.
    def store[F[_]: Sync](folder:Path): F[CommErr[Path]] = for { _ <- Sync[F].delay(folder.toFile.mkdirs()) fileName <- Sync[F].delay(UUID.randomUUID.toString + "_packet.bts") file = folder.resolve(fileName) fos <- Sync[F].delay(new FileOutputStream(file.toFile)) orErr <- Sync[F].delay { fos.write(packet.toByteArray) fos.flush() }.attempt resErr <- orErr match { case Left(th) => gracefullyClose(fos) *> Left(unableToStorePacket(packet, th)).pure[F] case Right(_) => gracefullyClose(fos) map { case Left(th) => Left(unableToStorePacket(packet, th)) case Right(_) => Right(file) } } } yield resErr Sample code
  • 12.
    def store(folder:Path): CommErr[Path]= { var fos: FileOutputStream = null try { folder.toFile.mkdirs() val fileName = UUID.randomUUID.toString + "_packet.bts" val file = folder.resolve(fileName) val fos = new FileOutputStream(file.toFile) fos.write(packet.toByteArray) fos.flush() gracefullyClose(fos) map { case Left(th) => Left(unableToStorePacket(packet, th)) case Right(_) => Right(file) } } catch { case ex:Exception => if(fos != null) gracefullyClose(fos) Left(unableToStorePacket(packet, th)) } } Sample code Rewrite
  • 17.
    • John Backus,1977 Turing Award lecture, “Can Programming Be Liberated from the von Neumann Style? A functional style and its algebra of programs” • Conventional programming languages are growing ever more enormous, but not stronger. Inherent defects at the most basic level cause them to be both fat and weak: their primitive word-at-a-time style of programming inherited from their common ancestor–the von Neumann computer, their close coupling of semantics to state transitions, their division of programming into a world of expressions and a world of statements, their inability to effectively use powerful combining forms for building new programs from existing ones, and their lack of useful mathematical properties for reasoning about programs. Von Neumann alive and still kicking
  • 18.
    Von Neumann • JohnBackus, 1977 Turing Award lecture, “Can Programming Be Liberated from the von Neumann Style? A functional style and its algebra of programs” • Conventional programming languages are growing ever more enormous, but not stronger. Inherent defects at the most basic level cause them to be both fat and weak: their primitive word-at-a-time style of programming inherited from their common ancestor–the von Neumann computer, their close coupling of semantics to state transitions, their division of programming into a world of expressions and a world of statements, their inability to effectively use powerful combining forms for building new programs from existing ones, and their lack of useful mathematical properties for reasoning about programs. alive and still kicking
  • 19.
  • 20.
  • 21.
    Functional Programming • Noorder of execution • No notion of time • Ordered independent evaluation • I/O - stateful and sequential
  • 22.
    Is monadic IOthe answer? • Conal Elliott, http: //conal.net/blog/ • Many publication in Haskell since 1995 • e.g.Functional Reactive Programming
  • 23.
    Is monadic IOthe answer?
  • 24.
    Is monadic IOthe answer? type IO a = RealWorld -> (a, RealWorld) main :: RealWorld -> ((), RealWorld) main :: RealWorld -> ((), RealWorld) main world0 = let (a, world1) = getChar world0 (b, world2) = getChar world1 in ((), world2)
  • 25.
    Is monadic IOthe answer?
  • 26.
    Is monadic IOthe answer?
  • 27.
    Speaking of IO •Did you know Haskell has exceptions? • Throwing exceptions is pure (!!!), function only depends on body + args • Caching them is not, so can be done only in IO Exceptions
  • 28.
    Speaking of IO forkIO:: IO () -> IO ThreadId throwTo :: Exception e => ThreadId -> e -> IO () timeout :: Int -> IO a -> IO (Maybe a) def forkIO(io:IO[Unit]): IO[ThreadId] def throwTo(threadId: ThreadId, Exception e): IO[Unit] def timeout[A](microseconds:Int, ioa: IO[A]): IO[Option[A]] Async Exceptions
  • 29.
    -- Executes anIO computation with asynchronous exceptions masked. -- That is, any thread which attempts to raise an exception in the -- current thread with throwTo will be blocked until asynchronous -- exceptions are unmasked again. mask :: ((forall a. IO a -> IO a) -> IO b) -> IO b Masking exceptions
  • 30.
    • Monad Transformers •MTL • Free Monads • Eff • Tagless final • ??? FP Techniques
  • 31.
    Monad Transformers trait Env traitAuditEvent type AuditLog = Vector[AuditEvent] trait Errors type ValidationError = NonEmptyList[Errors] type Metrics = Int type Capability[A] = ReaderT[IO, Env, A] type Audited[A] = WriterT[Capability, AuditLog, A] type Measured[A] = StateT[Audited, Metrics, A] type Action[A] = EitherT[Measured, ValidationError, A] def log(ae: AuditEvent): Action[Unit] = EitherT[ValidationError]( StateT.set( WriterT.put( ReaderT.ask((env:Env) => IO.pure({}))( Vector(ae)))))
  • 32.
    Monad Transformers Scala 
 Slow,lot of allocations, hard to use, unreadable Haskell
 WriterT
 space leaks StateT 
 exceptions
 thread-safety 
 put 4 >> concurrently (modify (+ 1)) (modify (+ 2)) >> get ReaderT - used widely Other: ResourceT, ExceptT, ClientT
  • 33.
    MTL trait Env trait AuditEvent typeAuditLog = Vector[AuditEvent] trait Errors type ValidationError = NonEmptyList[Errors] type Metrics = Int type Capability[A] = ReaderT[IO, Env, A] type Audited[A] = WriterT[Capability, AuditLog, A] type Measured[A] = StateT[Audited, Metrics, A] type Action[A] = EitherT[Measured, ValidationError, A] def log[F[_]: MonadState : MonadReader : MonadWriter: MonadError]: F[Unit] { val ms = MonadState[F] val md = MonadReader[F] val mw = MonadWriter[F] val me = MonadError[F] for { ( ...) } yield }
  • 34.
    MTL • IORef /Ref are becoming a backbone • MonadWriter
 PureScript WriterT -> IORef 10x mem reduction • MonadReader
 Useful for classy optics • MonadState
 IORef solves StateT problems
  • 35.
    sealed abstract classDockerOp[A] extends Product with Serializable object DockerOp { final case class Extract(unit: UnitDef) extends DockerOp[Image] final case class Tag(image: Image, registry: RegistryURI) extends DockerOp[(Int, Image)] final case class Push(image: Image) extends DockerOp[(Int, List[Docker.Push.Output])] final case class Pull(image: Image) extends DockerOp[(Int, List[Docker.Pull.Output])] type DockerF[A] = Free[DockerOp, A] } class Docker(cfg: DockerConfig, scheduler: ScheduledExecutorService, ec: ExecutionContext) extends (DockerOp ~> IO) { def apply[A](op: DockerOp[A]): IO[A] = op match { case DockerOp.Tag(i, r) => tag(i, i.to(r)).retryExponentially()(scheduler, ec) case DockerOp.Push(i) => push(i).retryExponentially()(scheduler, ec) case DockerOp.Pull(i) => pull(i).retryExponentially()(scheduler, ec) case DockerOp.Extract(unit) => extract(unit) } Free Monads Nelson
  • 36.
    final class KubernetesShell( kubectl:Kubectl, timeout: FiniteDuration, executionContext: ExecutionContext, scheduledES: ScheduledExecutorService ) extends (SchedulerOp ~> IO) final class NomadHttp( cfg: NomadConfig, nomad: Infrastructure.Nomad, client: org.http4s.client.Client[IO], scheduler: ScheduledExecutorService, ec: ExecutionContext ) extends (SchedulerOp ~> IO) sealed abstract class SchedulerOp[A] type SchedulerF[A] = Free[SchedulerOp, A] final case class Delete(dc: Datacenter, d: Datacenter.Deployment) extends SchedulerOp[Unit] final case class Launch(i: Image, dc: Datacenter, ns: NamespaceName, a: UnitDef @@ Versioned, p: Plan, hash: String) extends SchedulerOp[String] final case class Summary(dc: Datacenter, ns: NamespaceName, sn: Datacenter.StackName) extends SchedulerOp[Option[DeploymentSummary]] Free Monads Nelson
  • 37.
    trait FileAlg[F[_]] { defdeleteForce(file: File): F[Unit] def ensureExists(dir: File): F[File] def home: F[File] def removeTemporarily[A](file: File)(fa: F[A]): F[A] def readFile(file: File): F[Option[String]] def walk(dir: File): Stream[F, File] def writeFile(file: File, content: String): F[Unit] def writeFileData(dir: File, fileData: FileData): F[Unit] = writeFile(dir / fileData.name, fileData.content) } trait SbtAlg[F[_]] { def addGlobalPlugin(plugin: FileData): F[Unit] def addGlobalPlugins: F[Unit] def getDependencies(repo: Repo): F[List[Dependency]] def getUpdatesForProject(project: ArtificialProject): F[List[Update.Single]] def getUpdatesForRepo(repo: Repo): F[List[Update.Single]] } Tagless-final Scala Steward
  • 38.
    final case classContext[F[_]]( config: Config, dependencyService: DependencyService[F], fileAlg: FileAlg[F], filterAlg: FilterAlg[F], gitAlg: GitAlg[F], gitHubApiAlg: GitHubApiAlg[F], logger: Logger[F], nurtureAlg: NurtureAlg[F], processAlg: ProcessAlg[F], sbtAlg: SbtAlg[F], updateService: UpdateService[F], workspaceAlg: WorkspaceAlg[F] ) Tagless-final Scala Steward
  • 39.
    def run(args: List[String]):IO[ExitCode] = Context.create[IO](args).use { ctx => ctx.logger.infoTotalTime("run") { for { repos <- readRepos[IO](ctx.config.reposFile) _ <- prepareEnv(ctx) _ <- repos.traverse(ctx.dependencyService.forkAndCheckDependencies) allUpdates <- ctx.updateService.checkForUpdates(repos) reposToNurture <- ctx.updateService.
 filterByApplicableUpdates(repos, allUpdates) _ <- IO(reposToNurture.map(_.show).foreach(println)) _ <- IO(println(reposToNurture.size)) _ <- reposToNurture. filter(repos.contains).traverse_(ctx.nurtureAlg.nurture) //_ <- repos.traverse_(ctx.nurtureAlg.nurture) } yield ExitCode.Success } Tagless-final Scala Steward
  • 40.
    Domain Specific Languages traitStorageOp[A] object StorageOp { type StorageF[A] = Free[StorageOp, A] final case class Get(key:String) extends StorageOp[Option[String]] final case class Put(key:String, data:String) extends StorageOp[Unit] final case class Remove(key:String) extends StorageOp[Unit] def get(key: String): StorageF[Option[String]] = Free.liftF(Get(key)) def put(key: String, data: String): StorageF[Unit] = Free.liftF(Put(key, data)) def remove(key: String): StorageF[Unit] = Free.liftF(Remove(key)) } trait Storage[F[_]] { def get(key: String): F[Option[String]] def put(key: String, data: String): F[Unit] def remove(key: String): F[Unit] } def TFProgram[F[_]: Monad : Storage]: F[Unit] = { val storage: Storage[F] = implicitly import storage._ for { _ <- put("key1", "data1") data <- get("key1") _ <- remove("key1") } yield () } def FreeProgram: StorageF[Unit] = { import StorageOp._ for { _ <- put("key1", "data1") data <- get("key1") _ <- remove("key1") } yield () }
  • 41.
    Domain Specific Languages deffreeCompiler: StorageOp ~> IO = new (StorageOp ~> IO) { val mock = new MockStorage import mock._ def apply[A](fa: StorageOp[A]): IO[A] = fa match { case Put(key, value) => put(key, value) case Get(key) => get(key) case Remove(key) => remove(key) } } def TFCompiler: Storage[IO] = new Storage[IO] { val mock = new MockStorage import mock._ def get(key: String): IO[Option[String]] = get(key) def put(key: String, value: String): IO[Unit] = put(key, value) def remove(key: String): IO[Unit] = remove(key) } } class MockStorage { private val kvs = mutable.Map.empty[String, String] def get(key: String): IO[Option[String]] = { IO.pure(kvs.get(key)) } def put(key: String, value: String): IO[Unit] = { kvs(key) = value IO.unit } def remove(key: String): IO[Unit]= { kvs.remove(key) IO.unit } }
  • 42.
    def store[F[_]: Sync](folder:Path): F[CommErr[Path]] = for { _ <- Sync[F].delay(folder.toFile.mkdirs()) fileName <- Sync[F].delay(UUID.randomUUID.toString + "_packet.bts") file = folder.resolve(fileName) fos <- Sync[F].delay(new FileOutputStream(file.toFile)) orErr <- Sync[F].delay { fos.write(packet.toByteArray) fos.flush() }.attempt resErr <- orErr match { case Left(th) => gracefullyClose(fos) *> Left(unableToStorePacket(packet, th)).pure[F] case Right(_) => gracefullyClose(fos) map { case Left(th) => Left(unableToStorePacket(packet, th)) case Right(_) => Right(file) } } } yield resErr Sample code
  • 43.
  • 44.
    Layer 1 • TheReal-World layer • No business logic • Infra where created application run on • Logging, Metrics, Http client, Database, S3
  • 45.
    HTTP abstract class MonadHttpClient[F[_]]{ def get(url: Url): F[Data] def post[A: ToJson](url: Url, a: A): F[Data] }
  • 46.
    HTTP abstract class MonadHttpClient[F[_]]{ def get(url: Url): F[Data] def post[A: ToJson](url: Url, a: A): F[Data] }
  • 47.
    HTTP class ProdHttpClient extendsMonadHttpClient[IO] { def get(url: Url): IO[Data] = def post[A: ToJson](url: Url, a: A): IO[Data] = } Production
  • 48.
    HTTP abstract class MockHttp[A]{ self => (…) } Testing
  • 49.
    HTTP abstract class MockHttp[A]{ self => def map[B](f: A => B): MockHttp[B] = ??? def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ??? }
  • 50.
    HTTP abstract class MockHttp[A](env:???) { self => def map[B](f: A => B): MockHttp[B] = ??? def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ??? }
  • 51.
    HTTP abstract class MockHttp[A](env:HttpEnv) { self => def map[B](f: A => B): MockHttp[B] = ??? def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ??? } type HttpEnv = Id[HttpState] type HttpState = ConcurrentHashMap[Url, Data]
  • 52.
    HTTP abstract class MockHttp[A]{ self => def runMockHttp: ReaderT[IO, HttpEnv, A] = ??? def map[B](f: A => B): MockHttp[B] = ??? def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = ??? } type HttpEnv = Id[HttpState] type HttpState = ConcurrentHashMap[Url, Data]
  • 53.
    HTTP abstract class MockHttp[A]{ self => def runMockHttp: ReaderT[IO, HttpEnv, A] def map[B](f: A => B): MockHttp[B] = new MockHttp[B] { override def runMockHttp: ReaderT[IO, HttpEnv, B] = self.runMockHttp.map(f) } def flatMap[B](f: A => MockHttp[B]): MockHttp[B] = new MockHttp[B] { override def runMockHttp: ReaderT[IO, HttpEnv, B] = self.runMockHttp.flatMap(a => f(a).runMockHttp) } } type HttpEnv = Id[HttpState] type HttpState = ConcurrentHashMap[Url, Data]
  • 54.
    HTTP implicit def monadInstance:Monad[MockHttp] = ??? implicit def instance: MonadHttpClient[MockHttp] = ???
  • 55.
    HTTP implicit def monadInstance:Monad[MockHttp] = new Monad[MockHttp] { override def flatMap[A, B](fa: MockHttp[A])(f: A => MockHttp[B]): MockHttp[B] = fa.flatMap(f) override def pure[A](x: A): MockHttp[A] = new MockHttp[A] { override def runMockHttp: ReaderT[IO, HttpEnv, A] = ReaderT[IO, HttpEnv, A](_ => IO.pure(x)) } } implicit def instance: MonadHttpClient[MockHttp] = ???
  • 56.
    HTTP implicit def monadInstance:Monad[MockHttp] = new Monad[MockHttp] { override def flatMap[A, B](fa: MockHttp[A])(f: A => MockHttp[B]): MockHttp[B] = fa.flatMap(f) override def pure[A](x: A): MockHttp[A] = new MockHttp[A] { override def runMockHttp: ReaderT[IO, HttpEnv, A] = ReaderT[IO, HttpEnv, A](_ => IO.pure(x)) } } implicit def instance: MonadHttpClient[MockHttp] = new MonadHttpClient[MockHttp] { override def get(url: Url): MockHttp[Data] = new MockHttp[Data] { override def runMockHttp: ReaderT[IO, HttpEnv, Data] = ReaderT { env: HttpEnv => IO(env.getOrDefault(url, "404 NotFound")) } } override def post[A: ToJson](url: Url, a: A): MockHttp[Data] = new MockHttp[Data] { override def runMockHttp: ReaderT[IO, HttpEnv, Data] = ReaderT { env: HttpEnv => IO(env.put(url, implicitly[ToJson[A]].encode(a))) } } }
  • 57.
    Why MockHttp • Limitingyourself • Creating DSL • Not possible to use Sync[F] !!!
  • 58.
    Why MockHttp • Limitingyourself • Creating DSL • Not possible to use Sync[F] !!!
  • 59.
    Logging abstract class MonadLogging[F[_]]{ def info(s: String): F[Unit] def warning(s: String): F[Unit] } final case class LoggingConfig(context: Class[_]) abstract class JavaLogging[A] { self => def run: ReaderT[IO, LoggingConfig, A] def map[B](f: A => B): JavaLogging[B] = new JavaLogging[B] { override def run: ReaderT[IO, LoggingConfig, B] = self.run.map(f) } def flatMap[B](f: A => JavaLogging[B]): JavaLogging[B] = new JavaLogging[B] { override def run: ReaderT[IO, LoggingConfig, B] = self.run.flatMap(a => f(a).run) } } implicit val monadInstance = new Monad[JavaLogging] { override def pure[A](x: A): JavaLogging[A] = new JavaLogging[A] { override def run: ReaderT[IO, LoggingConfig, A] = ReaderT[IO, LoggingConfig, A](_ => IO.pure(x)) } override def flatMap[A, B](fa: JavaLogging[A])(f: A => JavaLogging[B]): JavaLogging[B] = fa.flatMap(f) } implicit def instance: MonadLogging[JavaLogging] = new MonadLogging[JavaLogging] { override def info(s: String): JavaLogging[Unit] = new JavaLogging[Unit] { override def run: ReaderT[IO, LoggingConfig, Unit] = ReaderT(x => IO(Logger.getLogger(x.context.getName).info(s))) } override def warning(s: String): JavaLogging[Unit] = new JavaLogging[Unit] { override def run: ReaderT[IO, LoggingConfig, Unit] = ReaderT(x => IO(Logger.getLogger(x.context.getName).warning(s))) } }
  • 60.
    Running def mockHttpRequest [F[_]:MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ??? def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ???
  • 61.
    Running def mockHttpRequest [F[_]:MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ??? def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ??? run :: forall f a. MonadHttpClient f => f a -> IO a def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A]
  • 62.
    Running def mockHttpRequest [F[_]:MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ??? def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ??? run :: forall f a. MonadHttpClient f => f a -> IO a def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A] Rank-N types
  • 63.
    Running def mockHttpRequest [F[_]:MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ??? def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ??? run :: forall f a. MonadHttpClient f => f a -> IO a def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A] Rank-N types run :: forall a. (forall f. MonadHttp f => f a) -> IO a
  • 64.
    Running def mockHttpRequest [F[_]:MonadHttpClient, A](http:HttpEnv)(req:F[A]): IO[A] = ??? def prodHttpRequests[F[_]: MonadHttpClient, A](req:F[A]): IO[A] = ??? run :: forall f a. MonadHttpClient f => f a -> IO a def run[F[_] : MonadHttpClient,A](ma:F[A]): IO[A] Rank-N types run :: forall a. (forall f. MonadHttp f => f a) -> IO a def run[A](ma:F[A] forAll {type F[_]}): IO[A]
  • 65.
    HttpService abstract class HttpService[F[_]]{ def runHttp[A](fa:F[A]): IO[A] }
  • 66.
    HttpService abstract class HttpService{ type F[_] def runHttp[A](fa:F[A]): IO[A] }
  • 67.
    HttpService abstract class HttpService{ type HttpEff[_] def runHttp[A](fa:HttpEff[A]): IO[A] }
  • 68.
    HttpService abstract class HttpService{ type HttpEff[_] val MH: MonadHttpClient[HttpEff] def runHttp[A](fa:HttpEff[A]): IO[A] }
  • 69.
    HttpService abstract class HttpService{ type HttpEff[_] val MH: MonadHttpClient[HttpEff] implicit val HttpEffM: Monad[HttpEff] def runHttp[A](fa:HttpEff[A]): IO[A] }
  • 70.
    Services abstract class Services{ val http: HttpService val logging: LoggingService val metrics: MetricsServic val s3Storage: StorageService } def mockServices(): Services = new Services { val httpEnv: HttpEnv = new ConcurrentHashMap() override val http: HttpService = new HttpService { override type HttpEff[A] = MockHttp[A] override implicit val a: Monad[HttpEff] = MockHttp.monadInstance override implicit val MH: MonadHttpClient[HttpEff] = MockHttp.instance override def runHttp[A](fa:MockHttp[A]): IO[A] = fa.runMockHttp.run(httpEnv) (…) }
  • 71.
    Application case class Application[A](unwrap:ReaderT[IO, Services, A]) { def run(services: Services): IO[A] = unwrap.run(services) } object Application { def apply[A](f: Services => IO[A]): Application[A] = Application(ReaderT[IO, Services, A](f)) implicit def monadInstance: Monad[Application] = new Monad[Application] { override def flatMap[A, B](fa: Application[A]) (f: A => Application[B]): Application[B] = new Application(fa.unwrap.flatMap { (a: A) => f(a).unwrap }) override def pure[A](x: A): Application[A] = Application(_ => IO.pure(x)) } } def logWarning(msg: String, ctx: Class[_] = this.getClass) = Application[Unit] { services: Services => import services.logging._ runLogging[Unit](ctx)( for { _ <- ML.warning(msg) } yield () ) } def downloadS3KeyFromWebService: Application[Data] = Application { services: Services => import services.http._ runHttp { for { _ <- MH.post("http: //xyz.com", Random.alphanumeric.take(Random.nextInt(15)).mkString) i <- MH.get("http: //xyz.com") } yield i } }
  • 72.
    object Main extendsIOApp { val program: Application[ExitCode] = for { key <- downloadS3KeyFromWebService _ <- logWarning(s"Key downloaded: ${key}") _ <- bumpCounterMetrics(IncCounter("Http", key.length)) m1 <- readCounter(ReadValue("Http")) _ <- logWarning(s"Number of bytes downloaded $m1") dh <- readFromS3(s"s3: //rpp/test/bucket/$key") bytesSaved <- storeOnLocalDisk("output.txt", dh) _ <- logWarning(s"Number of bytes saved $bytesSaved") } yield ExitCode.Success override def run(args: List[String]): IO[ExitCode] = { program.run(Services.mockServices()) } }
  • 73.
    Interpretation abstract class HttpService{ type HttpEff[_] def runHttp[A](fa:HttpEff[A]): IO[A] }
  • 74.
    Interpretation abstract class HttpService{ type HttpEff[_] def runHttp[A]: InterpreterFor[IO, HttpEff] } type InterpreterFor[F[_], G[_]] = FunctionK[G, F]
  • 75.
    Interpretation abstract class HttpService{ type HttpEff[_] def runHttp[A]: IO InterpreterFor HttpEff } type InterpreterFor[F[_], G[_]] = FunctionK[G, F]
  • 76.
    Interpretation abstract class HttpService{ type HttpEff[_] def runHttp[A]: IO InterpreterFor HttpEff } type InterpreterFor[F[_], G[_]] = FunctionK[G, F] abstract class Services { val http: IO InterpreterFor HttpEff val logging: IO InterpreterFor LogEff val metrics: IO InterpreterFor MetricEff val s3Storage: IO InterpreterFor StorageEff }
  • 77.
    Interpretation abstract class HttpService{ type HttpEff[_] def runHttp[A]: IO InterpreterFor HttpEff } type InterpreterFor[F[_], G[_]] = FunctionK[G, F] abstract class Services[G[_]:Sync] { val http: G InterpreterFor HttpEff val logging: G InterpreterFor LogEff val metrics: G InterpreterFor MetricEff val s3Storage: G InterpreterFor StorageEff }
  • 78.
    Interpretation abstract class HttpService{ type HttpEff[_] def runHttp[A]: IO InterpreterFor HttpEff } type InterpreterFor[F[_], G[_]] = FunctionK[G, F] abstract class Services { val http: Free[HttpF, ?] InterpreterFor HttpEff val logging: Free[LoggingF, ?] InterpreterFor LogEff val metrics: Free[MetricsF, ?] InterpreterFor MetricEff val s3Storage: Free[StorageF, ?] InterpreterFor StorageEff }
  • 79.
    Layer 2 • BusinessDomain Resources • Things that work with external world but for some purpose • Examples: UserRepository, Payment processor apis
  • 80.
    Layer 2 example traitAbExperiments[F[_]] { def getVersion(algoArea: String, tag: String): F[Version] }
  • 81.
    Layer 2 example traitAbExperiments[F[_]] { def getVersion(algoArea: String, tag: String): F[Version] } trait AlgoTrainer[F[_]] { def train(version: Version): F[Unit] }
  • 82.
    Layer 2 example traitAbExperiments[F[_]] { def getVersion(algoArea: String, tag: String): F[Version] } trait AlgoTrainer[F[_]] { def train(version: Version): F[Unit] } override def run(args: List[String]): IO[ExitCode] = { def trainProgram[F[_]: Monad](implicit algoTrainer: AlgoTrainer[F], abExperiments: AbExperiments[F]): F[Unit] = for { version <- abExperiments.getVersion("algoTag", "tag") _ <- algoTrainer.train(version) } yield () val trainProgramApp = trainProgram[Application].map(_ => ExitCode.Success) trainProgramApp.run(Services.mockServices()) }
  • 83.
    Layer 2 example implicitdef abExperiments: AbExperiments[Application] = (algoArea: String, tag: String) => Application { services => import services.http._ import services.logging._ for { _ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training")) v <- runHttp(MH.get(s"$algoArea/$tag")) } yield v } implicit def algoTrainApp: AlgoTrainer[Application] = (version: Version) => Application { services => import services.logging._ runLogging(AlgoTrainer.getClass)(ML.info(s"Training $version")) }
  • 84.
    Layer 2 example typeAppT[F[_], Ctx, A] = ReaderT[F, Ctx, A] type Getter[S, A] = S => A type Has[A, Ctx] = Getter[Ctx,A] implicit def abExperiments: AbExperiments[Application] = (algoArea: String, tag: String) => Application { services => import services.http._ import services.logging._ for { _ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training")) v <- runHttp(MH.get(s"$algoArea/$tag")) } yield v }
  • 85.
    Layer 2 example typeAppT[F[_], Ctx, A] = ReaderT[F, Ctx, A] type Getter[S, A] = S => A type Has[A, Ctx] = Getter[Ctx,A] implicit def abExperiments: AbExperiments[Application] = (algoArea: String, tag: String) => Application { services => import services.http._ import services.logging._ for { _ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training")) v <- runHttp(MH.get(s"$algoArea/$tag")) } yield v } implicit def abExperiments[F[_]: Monad, Ctx](implicit httpG: Has[HttpService[F], Ctx], loggingG: Has[LoggingService[F], Ctx]): AbExperiments[AppT[F, Ctx, ?]] = (algoArea: String, tag: String) => ReaderT[F, Ctx, Version] { (ctx: Ctx) => val http: HttpService[F] = httpG(ctx) val logging: LoggingService[F] = loggingG(ctx) import http._ import logging._ for { _ <- runLogging(AbExperiments.getClass)(ML.info("Getting data for training")) v <- runHttp(MH.get(s"$algoArea/$tag")) } yield v }
  • 86.
    Layer 3 • Businesslogic only • All pure functions • All data already obtained • No problem to use Option, Either and friends • Probably not type class based
  • 87.
  • 88.
    Ethics • No hardevidence that types work • Might be immutability • Clean room process • COCOMO: Upfront design reduces 10k defect by 10% • Coq • TLA+
  • 89.