Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scala Matsuri 2017

1,187 views

Published on

http://2017.scalamatsuri.org/

Published in: Engineering

Scala Matsuri 2017

  1. 1. Let's Build a Serverless Architecture in Scala! Yoshitaka Fujii (@yoshiyoshifujii)
 2017 ScalaMatsuri
 Saturday, February 25th
 16:30 - 17:10 Conference Room 1
  2. 2. Who am I ? • Yoshitaka Fujii (@yoshiyoshifujii) • Joined MOTEX in April, 2014 • Software Engineer • Scala(19 months) • Scala Kansai summit 2016 staff. 自己紹介。エムオーテックス株式会社。Scala歴19ヶ月。Scala関西サミット2016でスタッフさせていただきました。
  3. 3. Agenda • Serverless Architecture • Development & Deployment • DDD
  4. 4. Serverless Architecture • using FaaS • a significant reduction in costs • scalability • the lack of operating and maintaining infrastructure サーバレスアーキテクチャとは、FaaSを活用したシステム。コストの大幅な削減。スケーラビリティ。 インフラの運用管理が不要。
  5. 5. Function as a Service • AWS Lambda • Google Cloud Function • Microsoft Azure Functions
  6. 6. AWS Lambda • AWS Lambda is a compute service that lets you run code without provisioning or managing servers. • executes your code only when needed. • scales automatically, a few requests per day to thousands per second. • Node.js, Java, C# and Python AWS Lambda はサーバーをプロビジョニングしたり管理しなくてもコードを実行できるコンピューティングサービスです。必要な ときにコードを実行し、自動的にスケールします。
  7. 7. AWS Lambda Events • API Gateway (HTTP Requests) • Kinesis Streams • DynamoDB • S3 • Schedule (CloudWatch) • SNS • IoT これらのAWSサービスのイベントに応じてAWS Lambdaを実行できます。今回は、主にAPI GatewayとKinesis Streamsについて 取り上げます。
  8. 8. Amazon API Gateway • Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. • Low-Cost and Efficient.
 (only for calls made to your APIs and data transfer out) • Performance at Any Scale • Run Your APIs Without Servers フルマネージドサービスで、開発者はどのようなスケールであっても、簡単に API の作成、配布、保守、監視、保護が行えます。
  9. 9. https://aws.amazon.com/jp/api-gateway/details/
  10. 10. Amazon Kinesis Streams • Use Amazon Kinesis Streams to collect and process large streams of data records in real time. • rapid and continuous data intake and aggregation. • Accelerated log and data feed intake and processing. • Real-time metrics and reporting. • Real-time data analytics. • Complex stream processing. Amazon Kinesis Streams を使用して、データレコードの大量のストリームをリアルタイムで収集し、処理します。 高速かつ継続的 にデータの取り込みと集約を行うことができます。
  11. 11. https://docs.aws.amazon.com/ja_jp/streams/latest/dev/key-concepts.html
  12. 12. Amazon API Gatewayclient AWS Lambda Amazon S3 Amazon DynamoDB Amazon Kinesis AWS Lambda AWS Lambda Amazon Elasticsearch Service Context + Token Principal + Policy Policy is cached Denied 403 Allowed Auth function Consumers function System diagrams
  13. 13. Amazon API Gateway client AWS Lambda Amazon S3 Amazon DynamoDB Amazon Kinesis AWS Lambda Amazon Elasticsearch Service Policy is cached Large scale system client client client client client AWS Lambda AWS Lambda AWS Lambda AWS Lambda AWS Lambda AWS Lambda AWS Lambda AWS Lambda AWS Lambda AWS Lambda AWS Lambda 大規模なシステムを想定しています。
  14. 14. Large scale system • More functional requirements. • AWS Lambda wants to be a simple function. • 1 Lambda per method request.
 GET:/hello => getHello
 POST:/hello => postHello • develop and deploy efficiently 大規模なシステムは、機能要件が多い。AWS LambdaはシンプルなFunctionを保ちたい。1メソッドリクエストにつきLambdaを1 つ。また効率良く開発とデプロイをしたい。
  15. 15. Development & Deployment 実際に開発してデプロイするあたりを紹介します。
  16. 16. How to make simple AWS Lambda. 単純なAWS Lambdaの作り方を紹介します。
  17. 17. RequestStreamHandler.java public interface RequestStreamHandler { /** * Handles a Lambda Function request * @param input The Lambda Function input stream * @param output The Lambda function output stream * @param context The Lambda execution environment context object. * @throws IOException */ public void handleRequest(InputStream input, OutputStream output, Context context) throws IOException; } AWS Lambdaの公開インタフェースです。この中身を実装すると任意の処理が実行できます。
  18. 18. RequestHandler.java public interface RequestHandler<I, O> { /** * Handles a Lambda Function request * @param input The Lambda Function input * @param context The Lambda execution environment context object. * @return The Lambda Function output */ public O handleRequest(I input, Context context); } I/Oを任意の型で実装できますが、Java Beanなクラスもしくは、プリミティブな型のみになっており、Javaの制約を受けたりするの で、あまりオススメしません。
  19. 19. build.sbt lazy val root = (project in file(".")). settings( name := "aws-lambda-scala", organization := "com.example", scalaVersion := "2.12.1", libraryDependencies ++= Seq( "com.amazonaws" % "aws-lambda-java-core" % "1.1.0" ) )
  20. 20. project/plugins.sbt addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
  21. 21. src/main/scala/com/example/Hello.scala class HelloHandler extends RequestStreamHandler { @throws(classOf[java.io.IOException]) override def handleRequest(input: InputStream, output: OutputStream, context: Context): Unit = { val bytes = toByteArray(input) output.write(bytes) } def toByteArray(input: InputStream) = Stream.continually(input.read).takeWhile(_ != -1).map(_.toByte).toArray }
  22. 22. assembly $ sbt > assembly [info] Packaging .../target/scala-2.12/aws-lambda-scala-assembly-0.1.0-SNAPSHOT.jar
  23. 23. AWS CLI - lambda create function $ aws lambda create-function --region us-east-1 --function-name aws-lambda-scala --zip-file fileb://aws-lambda-scala-assembly-0.1.0-SNAPSHOT.jar --role arn:aws:iam::${AWS Account ID}:role/lambda_basic_execution --handler com.example.HelloHandler::handleRequest --runtime java8 --timeout 15 --memory-size 512
  24. 24. AWS CLI - lambda create function { "LastModified": "2017-01-02T12:34:56.789+0000", "FunctionName": "aws-lambda-scala", "Runtime": "java8", "Version": "$LATEST", "Role": "arn:aws:iam::${AWS Account ID}:role/lambda_basic_execution", "CodeSha256": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "Handler": "com.example.HelloHandler::handleRequest", "Timeout": 15, "Description": "", "MemorySize": 512, "FunctionArn": "arn:aws:lambda:us-east-1:${AWS Account ID}:function:aws-lambda-scala", "CodeSize": 5264477 }
  25. 25. AWS CLI - lambda invoke $ aws lambda invoke --region us-east-1 --function-name aws-lambda-scala --payload 123 --invocation-type RequestResponse /tmp/response { "StatusCode": 200 } 123
  26. 26. Amazon API Gatewayclient AWS Lambda Amazon S3 Amazon DynamoDB Amazon Kinesis AWS Lambda AWS Lambda Amazon Elasticsearch Service Context + Token Principal + Policy Policy is cached Denied 403 Allowed Auth function Consumers function System diagrams
  27. 27. Labor … • It is hard to run Lambda. • I will `assembly` one by one and upload it. • It is troublesome to configure the API Gateway on the console. Lambdaで実行するまでが大変。assemblyしてアップロードしてを繰り返さないとうまく動くか分からない。コンソール上でAPI Gatewayの設定を行うのは面倒。
  28. 28. sbt-aws-serverless plugin • sbt plugin to deploy code to Amazon API Gateway and AWS Lambda. • https://github.com/yoshiyoshifujii/sbt-aws-serverless 自前で開発したsbt-pluginを使います。API GatewayやLambdaにデプロイするpluginです。
  29. 29. serverless framework • To be honest this way is better. • However, blue green deployment is difficult. 正直、serverless frameworkを使うのが良いと思います。ただ、ブルーグリーンデプロイができない印象です。
  30. 30. https://martinfowler.com/bliki/BlueGreenDeployment.html
  31. 31. Amazon API Gateway Stage Stage Variables test env : test prod env : prod Deployment Resources Lambda ARN 1 /hello hello:${stageVariables.env}_1 AWS Lambda Alias Publish Versions dev $LATEST test_1 1 prod_1 1 Blue-Green deployment - 1st release Amazon DynamoDB Table account-test account-prod
  32. 32. Amazon API Gateway Stage Stage Variables test env : test prod env : prod Deployment Resources Lambda ARN 1 /hello hello:${stageVariables.env}_1 2 /hello hello:${stageVariables.env}_2 AWS Lambda Alias Publish Versions dev $LATEST test_1 1 prod_1 1 test_2 2 Blue-Green deployment - 2nd release test Amazon DynamoDB Table account-test account-prod
  33. 33. Amazon API Gateway Stage Stage Variables test env : test prod env : prod Deployment Resources Lambda ARN 1 /hello hello:${stageVariables.env}_1 2 /hello hello:${stageVariables.env}_2 AWS Lambda Alias Publish Versions dev $LATEST test_1 1 prod_1 1 test_2 2 prod_2 2 Amazon DynamoDB Table account-test account-production Blue-Green deployment - 2nd release
  34. 34. in this case • Release the test environment as it is. • Quickly return to the previous version. • A/B Testing. • Canary Release.
 https://martinfowler.com/bliki/CanaryRelease.html この方式なら、テストした環境をそのままリリースできる。すぐに前のバージョンに戻すことができる。A/Bテストや、カナリア・リ リースに使える。
  35. 35. Giter8 • https://github.com/yoshiyoshifujii/sbt-aws-serverless-ddd.g8 $ sbt new yoshiyoshifujii/sbt-aws-serverless-ddd.g8 name [My Something Project]: version [0.1.0-SNAPSHOT]: organization [com.example]: package [com.example]: Giter8でサンプルプロジェクトを公開しました。
  36. 36. Domain Driven Design
  37. 37. DDD - Hexagonal architecture • Implementing Domain-Driven Design - 2013/2/16 • Vaughn Vernon • Mitigate vendor lock-in. DDDのヘキサゴナルアーキテクチャを採用する。ベンダーロックインを緩和できる。
  38. 38. Dependency model Application Infrastructure Domain API 依存モデル。
  39. 39. Dependency model Application Infrastructure Domain API APIとApplication層は、ベンダーロックイン。ドメインは、再利用可能。インフラは一部再利用可能。 vendor lock-in
  40. 40. Amazon API Gatewayclient AWS Lambda Amazon S3 Amazon DynamoDB Amazon Kinesis AWS Lambda AWS Lambda Amazon Elasticsearch Service Context + Token Principal + Policy Policy is cached Denied 403 Allowed Auth function Consumers function System diagrams Make this part
  41. 41. build.sbt - dependsOn lazy val domain = (project in file("./modules/domain")). lazy val infraDynamo = (project in file("./modules/infrastructure/dynamodb")). dependsOn(domain). lazy val infraKinesis = (project in file("./modules/infrastructure/kinesis")). dependsOn(domain). lazy val appHello = (project in file("./modules/application/hello")). dependsOn(infraLambda, infraDynamo, infraKinesis). マルチプロジェクトの依存関係を依存モデルの通りに設定。
  42. 42. Project tree • Sources under modules. • application is a module of Lambda. • Make the Infrastructure as finely as possible. • Reduce module size. プロジェクト構成。ソースはmodulesディレクトリの配下に置く。applicationがLambdaのモジュールになる。Infrastructureは限 りなく小さく作る。Lambdaのモジュールサイズを小さくするため。
  43. 43. domain • POSO (Plain old Scala object) • Do not specify anything for libraryDependencies. domain層。POSOで作る。libraryDependenciesに何も指定しない。
  44. 44. Account case class AccountId(value: String) extends ValueObject[String] case class Account( id: AccountId, version: Option[Version], email: String, password: String, name: String) extends Entity[AccountId]
  45. 45. AccountRepository trait AccountRepository { def save(account: Account): Either[DomainError, Account] def get(id: AccountId): Either[DomainError, Option[Account]] def findBy(email: String): Either[DomainError, Option[Account]] def findAll: Either[DomainError, Seq[Account]] }
  46. 46. AccountEventPublisher case class AccountModified( accountId: AccountId, email: String, password: String, name: String) trait AccountEventPublisher { def publish(modified: AccountModified): Either[DomainError, AccountModified] }
  47. 47. infrastructure • implementation of Repository or Domain Event. • DI with Cake pattern. • It has no direct inheritance relation. infrastructure層。RepositoryやDomainEventの実装を持つ。Cake patternでDIされる。直接の継承関係を持たない。
  48. 48. AccountRepositoryOnDynamoDB trait AccountRepositoryOnDynamoDB { def save(account: Account): Either[DomainError, Account] = ??? def get(id: AccountId): Either[DomainError, Option[Account]] = ??? def findBy(email: String): Either[DomainError, Option[Account]] = ??? def findAll: Either[DomainError, Seq[Account]] = ??? }
  49. 49. AccountEventPublisherOnKinesis trait AccountEventPublisherOnKinesis { def publish(modified: AccountModified): Either[DomainError, AccountModified] = ??? }
  50. 50. application • Generate Lambda's module. • Dependency injection. application層。Lambdaのモジュールを生成する。依存性が注入する。
  51. 51. Base trait Base extends BaseStreamHandler { val accountRepository: AccountRepository val accountEventPublisher: AccountEventPublisher override def handle(input: Input): Try[String] = Try { JsObject( "message" -> JsString("hello world!!") ).compactPrint } }
  52. 52. App class App extends Base { override val accountRepository = new AccountRepository with AccountRepositoryOnDynamoDB override val accountEventPublisher = new AccountEventPublisher with AccountEventPublisherOnKinesis }
  53. 53. Amazon API Gatewayclient AWS Lambda Amazon S3 Amazon DynamoDB Amazon Kinesis AWS Lambda AWS Lambda Amazon Elasticsearch Service Context + Token Principal + Policy Policy is cached Denied 403 Allowed Auth function Consumers function System diagrams Make this part
  54. 54. build.sbt - dependsOn lazy val domain = (project in file("./modules/domain")). lazy val infraDomain = (project in file("./modules/infrastructure/domain")). dependsOn(domain). lazy val appAccountModified = (project in file(“./modules/application/accountmodified”)). dependsOn(infraLambdaConsumer, infraDomain). 依存関係は、API Gatewayとほぼ同じ。
  55. 55. Demo
  56. 56. sbt deploy prod $ sbt -DAWS_ACCOUNT_ID=<AWS Account ID> -DAWS_ROLE_ARN=arn:aws:iam::<AWS Account ID>:role/<Role NAME> -DAWS_BUCKET_NAME=<BUCKET NAME> > deploy v1 API Gateway created: xxxxxxxxxx API Gateway put: xxxxxxxxxx Lambda deployed: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth Lambda published: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth:1 Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth:prod Authorizer: auauau Lambda deployed: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello Lambda published: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello:1 Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello:prod_1 Create Deployment: {Id: dddddd,Description: 0.1.0-SNAPSHOT,CreatedDate: Sat Feb 25 12:34:56 JST 2017,}
  57. 57. Amazon API Gateway Stage Stage Variables test env : test prod env : prod Deployment Resources Lambda ARN 1 /hello hello:${stageVariables.env}_1 AWS Lambda Alias Publish Versions dev $LATEST test_1 1 prod_1 1 Blue-Green deployment - 1st release Amazon DynamoDB Table account-test account-prod
  58. 58. sbt deploy test $ sbt -DAWS_ACCOUNT_ID=<AWS Account ID> -DAWS_ROLE_ARN=arn:aws:iam::<AWS Account ID>:role/<Role NAME> -DAWS_BUCKET_NAME=<BUCKET NAME> > deploy test API Gateway created: xxxxxxxxxx API Gateway put: xxxxxxxxxx Lambda deployed: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth Lambda published: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth:2 Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth:test Authorizer: auauau Lambda deployed: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello Lambda published: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello:2 Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello:test_2 Create Deployment: {Id: dddddd,Description: 0.1.0-SNAPSHOT,CreatedDate: Sat Feb 25 12:34:56 JST 2017,}
  59. 59. Amazon API Gateway Stage Stage Variables test env : test prod env : prod Deployment Resources Lambda ARN 1 /hello hello:${stageVariables.env}_1 2 /hello hello:${stageVariables.env}_2 AWS Lambda Alias Publish Versions dev $LATEST test_1 1 prod_1 1 test_2 2 Blue-Green deployment - 2nd release test Amazon DynamoDB Table account-test account-prod
  60. 60. sbt deployCopy test to prod $ sbt -DAWS_ACCOUNT_ID=<AWS Account ID> -DAWS_ROLE_ARN=arn:aws:iam::<AWS Account ID>:role/<Role NAME> -DAWS_BUCKET_NAME=<BUCKET NAME> > deployCopy test prod Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth:prod Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello:prod_2 Stage: prod
  61. 61. Amazon API Gateway Stage Stage Variables test env : test prod env : prod Deployment Resources Lambda ARN 1 /hello hello:${stageVariables.env}_1 2 /hello hello:${stageVariables.env}_2 AWS Lambda Alias Publish Versions dev $LATEST test_1 1 prod_1 1 test_2 2 prod_2 2 Amazon DynamoDB Table account-test account-production Blue-Green deployment - 2nd release
  62. 62. sbt deployDev $ sbt -DAWS_ACCOUNT_ID=<AWS Account ID> -DAWS_ROLE_ARN=arn:aws:iam::<AWS Account ID>:role/<Role NAME> -DAWS_BUCKET_NAME=<BUCKET NAME> > deployDev dev API Gateway created: xxxxxxxxxx API Gateway put: xxxxxxxxxx Lambda deployed: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-auth:dev Authorizer: auauau Lambda deployed: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello Lambda Alias: arn:aws:lambda:us-east-1:aaaaaaaaaaaa:function:$name-app-hello:dev Create Deployment: {Id: ddddd,Description: 0.1.0-SNAPSHOT,CreatedDate: Sat Feb 25 12:34:56 JST 2017,}
  63. 63. sbt deployList $ sbt -DAWS_ACCOUNT_ID=<AWS Account ID> -DAWS_ROLE_ARN=arn:aws:iam::<AWS Account ID>:role/<Role NAME> -DAWS_BUCKET_NAME=<BUCKET NAME> > deployList dev ============================================================================================================ xxxxxxxxxx ============================================================================================================ | Stage Name | Last Updated Date | Deployment Id | Description | |----------------------|--------------------------------|-----------------|--------------------------------| | dev | Mon Feb 20 21:30:07 JST 2017 | qirn6v | null | | v1 | Mon Feb 20 21:23:41 JST 2017 | 8tb1zr | null | ===================================================================================== xxxxxxxxxx ===================================================================================== | Created Date | Deployment Id | Description | |--------------------------------|-----------------|--------------------------------| | Mon Feb 20 21:30:07 JST 2017 | ddddd2 | 0.1.0-SNAPSHOT | | Mon Feb 20 21:23:41 JST 2017 | ddddd1 | 0.1.0-SNAPSHOT |
  64. 64. sbt invoke $ sbt -DAWS_ACCOUNT_ID=<AWS Account ID> -DAWS_ROLE_ARN=arn:aws:iam::<AWS Account ID>:role/<Role NAME> -DAWS_BUCKET_NAME=<BUCKET NAME> > invoke prod ============================================================ GET:https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com/prod/hellos ============================================================ 200 {"message":"hello world!!"}
  65. 65. Summary • Scala is easy to combine DDD and Serverless. • Easy to share processing with multiple projects. • Cheap, scalable and easy to operate. ScalaはDDDとServerlessを組み合わせるとやりやすい。マルチプロジェクトで処理を共通化しやすい。安くてスケーラブルで運用 が楽。

×