Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Airframe Meetup #3: 2019 Updates & AirSpec

236 views

Published on

Presentation slides of Airframe Meetup #3 https://airframe.connpass.com/event/148169/
- Airframe 19 Milestone
- AirSpec: A new testing library for Scala

-

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Airframe Meetup #3: 2019 Updates & AirSpec

  1. 1. Taro L. Saito, Ph.D. Arm Treasure Data October 23nd, 2019 Arm Treasure Data Tokyo Office Airframe Meetup #3 1
  2. 2. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. About Me: Taro L. Saito (Leo) 2 ● Principal Software Engineer at Arm Treasure Data ● Building distributed query engine services ● Living in US for 4 years ● DBMS & Data Science Background ● Ph.D. of Computer Science ● OSS Projects around Scala ● sbt-sonatype: used for releasing 3000+ Scala projects ● snappy-java: a compression library used in Spark, Parquet, etc. ● etc.
  3. 3. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Recent Work: Designing Data-Intensive Applications ● Techniques and concepts around distributed data processing systems 3
  4. 4. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Today’s Agenda ● Airframe Updates ● The current version: 19.10.1 ■ 25+ releases since July 2019~ ● AirSpec: a new testing framework for Scala ● Lifecycle design ● airframe-http updates ■ Route filters ● Releasing OSS with GitHub Actions ● sbt-sonatype: Blazing Fast Release to Sonatype ● 3 Lightning Talks 4 Airframe
  5. 5. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe: Collection of Our Best Practices ● Gather the best practices of Scala into Airframe OSS ● Get the real experiences by operating 24/7 services 5 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe
  6. 6. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: A New Testing Framework for Scala and Scala.js ● Simple Usage ● import wvlet.airspec._ ● Extend AirSpec ● Define test cases as public functions ● Low Learning Cost ● No annotation (like JUnit5) is necessary. ● Simple assertions: assert(cond), x shouldBe y, etc. ■ No need to learn other complex DSLs. 6
  7. 7. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Assertions 7
  8. 8. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Example: AirSpec Assertions 8
  9. 9. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Scala.js ● ScalaTest RefSpec ● Has similar syntax with AirSpec, but it doesn’t support Scala.js because of lack of runtime reflection. ● AirSpec uses airframe-surface to collect test methods ■ Scala.js: Collect test methods (MethodSurface) at compile-time ■ Scala JVM: Uses reflection at runtime 9
  10. 10. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Writing Specs in Natural Languages 10
  11. 11. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Motivation ● Q: Why do you need a new testing framework? ● A: For maintaining Airframe ● Because of (relatively) slow ScalaTest releases, we couldn’t catch up with the latest versions of Scala and Scala.js. ■ We usually needed to wait for several weeks or years (!) to test new Scala versions. ■ ScalaTest still doesn’t support Scala.js 1.0.x ● A: Taking a balance between minimalist approaches and too complex DSLs ● minimalist: minitest, scala-verify ■ Simple, but using them is not so fun (e.g., no support for test filtering by name) ● complex DSLs: ScalaTest, Specs2, uTest ■ High-learning cost ■ Need an agreement on the assertion style within the team 11
  12. 12. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Motivation ● Another answer: It was surprisingly simple to implement a testing framework 12
  13. 13. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec Features ● Production Ready ● All Airframe test cases are now using AirSpec ● sbt integration ● Handy keyword search for test methods ■ > testOnly -- (a pattern for class or method names) ● Property-based testing integrated with ScalaCheck ● Scala 2.11, 2.12, 2.13, and Scala.js (0.6.x and 1.0.x) support ● Integration with Airframe ● Automatic DI session management ● handy logging with airframe-log 13
  14. 14. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Logging with airframe-log 14
  15. 15. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec + DI ● Injecting dependency as test function arguments ● Override AirSpec.design to customize your designs ● Managing lifecycle of test dependencies ● The lifecycle (e.g., start and shutdown) of the injected services can be managed by Airframe DI ● Internally nested Airframe sessions will be created for individual test cases ● Nest-free resource management ● no loan pattern ● no monad 15 Lifecycle Design (New Feature!)
  16. 16. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Pro Tips: AirSpec + Airframe DI ● Tips ● Use objects to define component designs ● Add objects for test modules and override existing bindings (if necessary) ● Use the designs inside AirSpec ● Why? ● Your design becomes reusable for both production/test code ● Airframe DI uses lazy binding by default, so having unnecessary binding is OK ■ If necessary, use coverage report option to minimize the design 16
  17. 17. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. DI Coverage Stats 17
  18. 18. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Property-Based Testing with ScalaCheck 18
  19. 19. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe DI Update: Lifecycle Design ● AutoCloseable (parent of Closeable interface) support ● If an injected instance implements AutoCloseable interface (def close()), it will be automatically closed upon Session.shutdown ● Support defining lifecycle hooks within a design ● Useful for reusing services with a proper lifecycle management ● e.g., http client/server, database connection pool, etc. ● Testing environments can be prepared by adding necessary designs: ■ design + service1Design + service2Design + ... 19
  20. 20. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Example: Test Design 20
  21. 21. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-http: Using Scala As An IDL ● IDL: Interface Definition Language ● e.g. ProtocolBuffers, gRPC ● Use Scala’s Standard Functionalities ● case classes ■ Define HTTP request/response data structure ● functions ■ Define HTTP endpoints ■ Use @Endpoint annotation for for URL to method mapping ● Router ● Router.add[X] ● filter andThen router (Added in 19.8.x) 21
  22. 22. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Path Mapping ● :param ● Mapping to a function argument ● /user/:id ■ /user/1 ■ def getUser(id:Int) ○ getUser(1) ● *param ● Mapping url path tail to a function argument ● /resource/*path ■ /resource/my/key/path ■ def getPath(path:String) ○ getPath(my/key/path) ● case classes ● request/response body ● JSON <-> MessagePack <-> Object 22
  23. 23. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: airframe-http: Request Filter ● Router ● .add[X] ● .andThen[X] ● Filter ● apply(request, context) ■ returns Future[Response] 23
  24. 24. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-http-client ● Read server responses as target object types ● Supports both JSON and MessagePack ● Added requestFilter argument to manipulate HTTP headers 24
  25. 25. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Distributed Tracing with DataDog APM ● Setup ● Just run JVM applications with -javaagent:dd-agent.jar ● Implicit logging ● No explicit instrumentation is necessary ■ ByteBuddy is used to inject instrumentation code ● Major libraries are already supported ■ https://github.com/DataDog/dd-trace-java/tree/master/dd-java-agent/instru mentation ■ e.g., jetty, netty4, akka, okhttp, aws-sdk, servlet, spring, JDBC connections, etc. ● Cost: $31/instance/month, $1.2 / 1M tracing events 25
  26. 26. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. DataDog APM: Request Summary 26
  27. 27. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe/AirSpec 19 Milestones ● Airframe 19 Milestone ● https://github.com/wvlet/airframe/issues/528 ■ airframe-json: ○ Stream JSON -> MessagePack converter (done) ■ airframe-codec: Scala.js support ■ airframe-control: Adding circuit breaker ■ airframe-http: production ready ○ We have migrated several internal APIs into airframe-http. Advanced features like request logging, CORS support, admin interface, JMX metrics are internally available. Need to extract them. ■ airframe-http-client: Jetty, akka-http support ● AirSpec 19 Milestone ● https://github.com/wvlet/airframe/issues/606 ■ Standalone launcher and async testing ■ Using the return values of test functions for reporting ■ IntelliJ support (?) 27
  28. 28. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [WIP] Scala.js RPC ● Scala.js ● Compiling Scala code into JavaScript for Web Browsers ● airframe-codec: Passing model class data between Scala and Scala.js UserInfo MessagePack UserInfo Pack Unpack PackUnpack Scala Server Side Scala.js Client Side XML RPC 28 airframe-codec airframe-codec
  29. 29. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Releasing Airframe ● Migrated from Travis CI to GitHub Actions ● Uses .github/workflows folder ● Examples: https://github.com/wvlet/airframe/tree/master/.github/workflows ● Good integration with the pull request UI 29
  30. 30. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. GitHub Actions ● Keywords ● workflow -> job -> steps ● workflow ■ YAML file ■ define conditions to trigger jobs (PR, push, etc.) ● job ■ A unit of execution ○ = a single docker container ■ Multiple jobs run in parallel ■ No local folder sharing between jobs ● steps ■ Can import third-party step definitions ■ Define a sequence of command lines 30
  31. 31. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. GitHub Actions: Secrets ● Secrets (e.g., password, token) can be defined in GitHub settings ● Secrets cannot be used in PRs nor forked repositories ● exception: GITHUB_TOKEN ■ e.g., for updating PR tags, comments, etc. ● Uploading test artifacts (e.g., test coverage report) in PR to other than GitHub is not straightforward 31
  32. 32. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. GitHub Actions: Releasing to Sonatype ● Steps ● Import GPG private key ● publishSigned ■ sbt-pgp ● Upload to Sonatype ● Tips ● Needed to Install GPG1 ■ TTY env of GitHub action is broken, so gpg2-agent cannot pass PGP_PASSPHRASE to GPG2 ■ This might be a bug of sbt-pgp plugin ● Encode GPG private key with base64 and set it as secret.PGP_SECRET ■ decrypt with secret.PGP_PASSPHRASE 32
  33. 33. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● Blog post: https://medium.com/@taroleo/sbt-sonatype-f02bdafd78f1 ● Airframe needs to upload 2000+ files for each release ● When Sonatype API is slow, it could take 20 minutes ~ 1 hour ● With bundle upload, it takes less than 1 minute ● Bundle Upload ● publishTo := target/sonatype-bundle/(version) ■ Collecting artifacts to the local folder ● Upload artifacts as a single jar sbt-sonatype: Blazing Fast Upload to Sonatype 33
  34. 34. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Happy reports from sbt-sonatype users 34
  35. 35. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. scala-steward ● A time saver for OSS maintainers ● Create PRs for library updates ● Became the top committer for October 2019 35
  36. 36. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Summary ● AirSpec ● A new testing framework with DI support ● Define reusable designs ■ lifecycle of servicers can be designed now ● Airframe ● airframe-http is production-ready with filters ● Development ● Using GitHub Actions for CI and releasing Airframe ● Bundle upload with a new sbt-sonatype plugin 36
  37. 37. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Join Arm Treasure Data Team! ● Solving Challenges In The Real World Data Processing ● Building a scalable data processing platform on the cloud ● Advanced data analytics with Presto, Hive, Spark, and workflows ■ Processing 1 million queries / day ● Providing distributed computing as a service for 450+ companies 37 Airframe
  38. 38. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos! 38
  39. 39. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Module Mix-InPackaging HTTP Requests and Responses Data airframe-launcher > _ airframe-log production: port: 10010 user: xxxx ... airframe-config airframe-codec sbt-pack airframe-fluentd Scala Objects Table Data (CSV, TSV) JSON airframe-jsonairframe-surface airframe-tablet airframe-jmx Monitor Runtime States Generate Mapping Codec Metrics & Log Data JDBC ResultSets airframe-jdbc airframe-http airframe-http-finagle Launch HTTP Services airframe DI Debug Logs Schema-On-Read Mapping Airframe
  40. 40. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. The Architecture of Arm Treasure Data 40 DataLogs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing 2 million records / sec. 130 trillion records 1 billion rows processed / sec. Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS

×