Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018

359 views

Published on

Airframe is a collection of lightweight libraries useful for building full-fledged applications in Scala.

Published in: Engineering
  • Paid To Write? Earn up to $200/day on with simple writing jobs. ●●● http://t.cn/AieXS62G
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018

  1. 1. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Taro L. Saito, Ph.D. GitHub: @xerial Arm Treasure Data Airframe: Lightweight Building Blocks for Scala November 16th, 2018 Scale By The Bay 2018 - Unconference 1
  2. 2. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. About Me: Taro L. Saito (Leo) ● An Engineer with Research Background ● Ph.D., University of Tokyo ● DBMS & Genome Science ● Leading Query Engine Team ● Active OSS Developer ● airframe ● sqlite-jdbc ■ More than 1000 GitHub stars ● snappy-java ■ Compression library used in Spark, Parquet ● sbt-sonatype ■ Used in 2000+ Scala projects ● ... 2
  3. 3. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe ● Lightweight Building Blocks for Scala ● Essential libraries for building any applications ● Used in production for 2+ years ● Based on my code collection since 2009 ● Initially written in Java ● Gradually migrated to Scala ● Repackaged into wvlet.airframe in 2016 ● For maintainability ● 18 Modules ● Simplifying your daily programming in Scala 3
  4. 4. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. 18 Airframe Modules ● Bootstrap ● airframe-config Configuration loader ● airframe-launcher Command-line program launcher ● Object Serialization ● airframe-codec encoder/decoder SPI + standard codecs ● airframe-msgpack pure-Scala MessagePack implementation ● airframe-tablet CSV/TSV/JSON/JDBC ResultSet <-> Object ● Monitoring & Debugging ● airframe-log Logging ● airframe-metrics Human-readable metrics for time, date, data size, etc ● airframe-jmx Object metrics provider through JMX ● Building Service Objects ● airframe Dependency injection ● airframe-surface Object type inspector ● Misc: ● airframe-control, airframe-jdbc, airframe-json, airframe-http, etc. 4
  5. 5. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Utilities for Debugging Applications 5
  6. 6. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Debugging Applications: Airframe Log ● Airframe Log: A Modern Logging Library for Scala (Medium Blog) ● ANSI color, source code location support 6
  7. 7. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Metrics ● Human Readable Data Format (Duration, DataSize, etc.) ● Handy Time Window String Support 7 “-1d” “-1w” “-7d”
  8. 8. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe JMX ● Checking the internal states of remote JVM processes ● JMX clients ● jconsole has JMX metric monitor ● Airframe JMX -> DataDog -> Dashboard 8
  9. 9. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Launcher ● Using Class & Function Command Line Programs 9
  10. 10. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Dependency Injection with Airframe 10
  11. 11. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Today’s Goals ● Lean How to Use Airframe DI (Dependency Injection) ● Understand what can be simplified with DI ● Learn 5 Airframe DI Design Patterns ● That improve the thought processes in programming 11
  12. 12. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. What is Dependency Injection (DI)? ● Many Articles… ● Inversion of Control Containers and the Dependency Injection pattern. Martin Fowler (2004) ● StackOverflow, Wikipedia, … ● Many Frameworks... ● Spring, Google Guice, Scaldi, Macwire, Grafter, Weld, etc. ● No framework approaches also exist (Pure-Scala DI) ● Recent Definition: ● Dependency Injection is the process of creating the static, stateless graph of service objects, where each service is parameterised by its dependencies. ■ What is Dependency Injection? by Adam Warski (2018) ● However, it’s still difficult to understand what is DI 12
  13. 13. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Simplifying DI with Airframe ● Airframe Usage ● import wvlet.airframe._ ● Simple 3 Step DI ● bind ● design ● build 13
  14. 14. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Design Patterns In Airframe ● Pattern 0 (basic): Building Service Objects with Auto-Wiring ● Pattern 1: Configuring Modules ● Pattern 2: Switching Bindings for Tests ● Pattern 3: Managing Lifecycle in FILO order ● Pattern 4: Bundling Service Traits ● Flower-bundle pattern ● Pattern 5: Binding Factory 14
  15. 15. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Constructor vs In-Trait Injections ● Constructor Injection ● In-Trait Injection 15
  16. 16. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Building Service Objects ● When coding A and B ● You can focus on only direct dependencies ● You can forget about indirect dependencies ● Airframe DI builds A, B, and direct/indirect dependencies on your behalf. A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 16 You can forget this part
  17. 17. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Code Example ● Focus on Logic 17 A DB Client Fluentd Logger B
  18. 18. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Hand Wiring vs Auto Wiring 18
  19. 19. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Configuring Modules ● How to pass configuration objects to corresponding modules? ● new A(new DBClient(new ConnectionPool(new DB(new DBConfig(...)), new ConnectionPoolConfig(...), new DBClientConfig(...))) ● Things You Need to Remember ● Argument orders (or argument names in Scala) of individual modules ● How to instantiate modules A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 19 DB Config Connection Pool Config HttpClient Config
  20. 20. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 1: Adding Config ● Put config to the closest place in the code 20
  21. 21. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 2: Switching Bindings For Testing ● In Airframe Design ● You can replace DB and FluentdLogger to In-Memory Impl ● How to build A and B differs, but the same code can be used A Memory DB Connection Pool DB Client DB Monitor Fluentd Logger In-memory Logger B 21 Overriding Design for Testing
  22. 22. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 2: Switching Bindings For Testing 22
  23. 23. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 3: Managing Lifecycle ● onStart, onShutdown hooks ● JSR-250 annotations 23
  24. 24. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Complex FILO Order Resource Management ● FILO := First-In Last-Out ● Airframe registers onStart and onShutdown lifecycle hooks when creating instances ● When closing sessions, onShutdown will be called in the reverse order ● Dependencies forms DAG ● Dependencies will be generated when creating new service objects A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 134 56 7 2 8 Shared Resource 24
  25. 25. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 4: Bundling Service Traits ● Flower-Bundle Pattern ● Create composable services 25
  26. 26. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 5: Bind Factory ● Create a factory that change partial dependencies ● e.g., creating differently configured instances of Databases 26
  27. 27. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. 3 Things You Can Forget With Airframe DI ● 1. How to Build Service Objects ● config, auto-wiring, flower bundle pattern ● 2. How to Manage Resource Lifecycle ● FILO order ● 3. How to Use DI Itself (!!) ● Only need to understand bind, design, and build. 27
  28. 28. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Summary: Reducing Code Complexity with Airframe DI ● You can effectively forget about: ● How to build service objects ● How to manage resources in FILO order ● How to use DI itself A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 134 56 7 2 8 28 Implementation Details
  29. 29. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Internals 29
  30. 30. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Details ● bind[X] ● X becomes singleton ● Usually there is no need to declare bind[X].toSingleton ■ Unless you want to initialize X early with production mode ● bindInstance[X] ● When you need to call new X(d1, d2, …) every time ● bindFactory[A1 => X] ● For customizing A1 in X ● Design ● Immutable & Serializable ● design1 + design2 + …. ■ Overriding the previous design (adding order is important) ● Session ● withLazyMode/withProductionMode ● noLifeCycleLogging 30
  31. 31. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Injecting Airframe Session with Scala Macros 31
  32. 32. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Surface: Object Shape Inspector ● Reading Type Signatures From ScalaSig ● Scala compiler embeds Scala Type Signatures (ScalaSig) to class files ● Surface supports ● type alias, tagged type ● higher-kinded types class A (data:List[B]) class A data: List[java.lang.Object] class A data: List[java.lang.Object] ScalaSig: data:List[B] javac scalac Surface.of[A] data: List[B] scala.reflect.runtime. universe.TypeTag Type Erasure 32 ???
  33. 33. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Future Work & Summary 33
  34. 34. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Current State of Airframe ● Version 0.73 (As of November 2018) ● We already had 40+ releases in 2018 ● Automated Release ● Cross building libraries for Scala 2.11, 2.12, 2.13, and Scala.js ● ‘sbt release’ command took 3 hours ■ Sequential steps: ○ compile -> test -> package -> upload x 18 modules x 4 Scala versions ● Now a new version can be released in 10 minutes on Travis CI ● Blog ● 3 Tips for Maintaining Scala Projects ● Future Work ● Adding child session support ● Support multiple constructor argument blocks 34
  35. 35. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Philosophy: Simplicity By Design ● “Simplicity” by Philippe Dufour ● A clock made by a legendary watchmaker in Switzerland ● Every part of the clock is built by himself ● Airframe ● Provides simplicity for application developers 35
  36. 36. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Summary ● To understand DI, think about what you can simplify ● How to build objects ● How to manage resources (FILO) ● Learning DI framework itself ● 5 Airframe Design Patterns ● Pattern 1: Configuring Modules ● Pattern 2: Overriding Bindings for Tests ● Pattern 3: Managing Lifecycle in FILO order ● Pattern 4: Bundling Service Traits ■ Flower-bundle pattern ● Pattern 5: Binding Factory Don’t Forget Adding GitHub Star! wvlet/airframe 36
  37. 37. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos! 37

×