Successfully reported this slideshow.
Your SlideShare is downloading. ×

Scala 3, what does it means for me? / Scala 3って、私にはどんな影響があるの? by Joan Goyeau

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
Lt 111119
Lt 111119
Loading in …3
×

Check these out next

1 of 79 Ad
Advertisement

More Related Content

Similar to Scala 3, what does it means for me? / Scala 3って、私にはどんな影響があるの? by Joan Goyeau (20)

More from scalaconfjp (20)

Advertisement

Recently uploaded (20)

Scala 3, what does it means for me? / Scala 3って、私にはどんな影響があるの? by Joan Goyeau

  1. 1. JOAN GOYEAU SENIOR DATA ENGINEER Scala 3, What does it means to me? Scala 3 について話す前に、私の仕事・チームを紹介します。
  2. 2. “Netflix is just streaming movies. I don’t see what else you are doing there.” — Most of my friends Netflix って動画ストリーミングでしょ。他になにをやってるのかよくわからない、と多 くの友人から言われます。
  3. 3. その友人にとって Netflix はこんな感じ。
  4. 4. Internet Traffic The Global Internet Phenomena Report October 2018 Netflix はインターネット・トラフィック全体の15% を占めています。
  5. 5. Peak Time Internet Traffic in USA The Global Internet Phenomena Report October 2018 ピークタイムには、40% の占有率です。
  6. 6. ISP like Netflix のサービスはすべてAWS上にホストされています。みなさんは、docomo のよう なISPを通じてそれらにアクセスしているでしょう。
  7. 7. ISP like 😬 😠 💥 ISP, Netflix の双方が多大な転送コストを負っています。ISPにとって、トラフィック急 増をコストを負担してサポートすることは、好ましいことではないでしょう。
  8. 8. Open Connect ISP like 😉 😀 そのためISPの内側に、サーバーを配置します。双方にとって、転送コストの節約にな ります。ユーザーから見ると、レイテンシが大幅に削減されます。
  9. 9. High Quality Start Fast Minimize Rebuffer 再生開始までのレイテンシ、画質、バッファリング、はCAP定理のような三すくみの関 係です。
  10. 10. Data Driven データ・ドリブンな決定。そのためにとにかく、ABテストを行っています。
  11. 11. A B Baseline 21% Variation 26% 例えば、100万人ずつのグループに対してUI変更のABテストを実施しています。同様の スケールで、並行に多くのテストを行っています。
  12. 12. 1.6 Gb 4.6 Gb 1.86 Gb448 Mb720p 1080p Quality 2時間の動画を圧縮するエンコーダーを改良することができました。
  13. 13. 1.6 Gb 4.6 Gb 1.86 Gb448 Mb720p 1080p Quality イメージ画像は、トレーラーに対してもABテストを行います。
  14. 14. 別のテストの事例です。問題が起きたらフェイルオーバーされるべきだと誰かが言った ら、私達は「OK. 本番環境で試してみよう」といいます。
  15. 15. Scala 3, What does it means to me? Netflix では、大量のデータの処理に Scala を使っています。Scala 3 も楽しみにしてますが、私た ちユーザーのコーディングや問題解決にどのような影響を与えるか理解することは難しいことでし ょう。そこで、Scala 2 の悩み事がどのように解決されるか紹介します。
  16. 16. SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU 1. 2. 3. 4. 5. Episode 1: Limited Enumerations 制約のある列挙型
  17. 17. object PlayTraceType extends Enumeration { type PlayTraceType = Value val Start, Pause, Stop = Value } PlayTraceType.values PlayTraceType.withName("Pause") 全ての値の取得と、withName関数がある点は良いですね。
  18. 18. sealed trait PlayTraceType object PlayTraceType { case object Start extends PlayTraceType case object Pause extends PlayTraceType case object Stop extends PlayTraceType }
  19. 19. sealed abstract class PlayTraceType(description: String) object PlayTraceType { case object Start extends PlayTraceType("Start event") case object Pause extends PlayTraceType("Pause event") case object Stop extends PlayTraceType("Stop event") } PlayTraceType.values PlayTraceType.withName("Pause")
  20. 20. import enumeratum._ sealed abstract class PlayTraceType(description: String) extends EnumEntry object PlayTraceType extends Enum[PlayTraceType] { case object Start extends PlayTraceType("Start event") case object Pause extends PlayTraceType("Pause event") case object Stop extends PlayTraceType("Stop event") val values = findValues } PlayTraceType.values PlayTraceType.withValue("Pause")
  21. 21. enum PlayTraceType { case Play, Pause, Stop } PlayTraceType.enumValues PlayTraceType.enumValueNamed("Play")
  22. 22. enum PlayTraceType(description: String) { case Play extends PlayTraceType("Start event") case Pause extends PlayTraceType("Pause event") case Stop extends PlayTraceType("Stop event") } PlayTraceType.enumValues PlayTraceType.enumValueNamed("Play")
  23. 23. Episode 2: Either and Coproducts SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU Eitherと余積(Coproduct)
  24. 24. def audio(movieId: String): Either[NotFoundException, Array[Byte]] = ??? // Get audio here def subtitle(movieOffset: Int): Either[Exception, String] = if (movieOffset < 0) new IllegalArgumentException(s"$movieOffset is < than 0") else ??? // Get subtitles here. May throw NotFoundException
  25. 25. def audio(movieId: String): Array[Byte] | NotFoundException = ??? // Get audio here def subtitle(movieOffset: Int): String | IllegalArgumentException | NotFoundException = if (movieOffset < 0) new IllegalArgumentException(s"$movieOffset is < than 0") else ??? // Get subtitles here. May throw NotFoundException
  26. 26. def audio(movieId: String): Array[Byte] | NotFoundException = ??? // Get audio here def subtitle(movieOffset: Int Refined Positive): String | NotFoundException = ??? // Get subtitles here. May throw NotFoundException
  27. 27. Episode 3: nulls SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU nulls
  28. 28. val movie: Movie = null movie.age 💥 NullPointerException
  29. 29. val movie: Option[Person] = None movie.map(_.age)
  30. 30. val movie: Movie = null movie.age
  31. 31. val movie: Movie | Null = null movie.age
  32. 32. Episode 5: Tuple deconstruction in function SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU 関数におけるタプルの分解
  33. 33. Option(("Hello", "world")) .map { case (first, second) => s"$first $second" }
  34. 34. Option(("Hello", "world")) .map { (first, second) => s"$first $second" }
  35. 35. Episode 6: extends AnyVal SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU AnyValの継承
  36. 36. def recordView(movieId: String, userId: String) = ??? val userId = "some_user_id" val movieId = "some_movie_id" recordView(userId, movieId)
  37. 37. type MovieId = String type UserId = String def recordView(movieId: MovieId, userId: UserId) = ??? val userId: UserId = "some_user_id" val movieId: MovieId = "some_movie_id" recordView(userId, movieId)
  38. 38. case class MovieId(value: String) case class UserId(value: String) def recordView(movieId: MovieId, userId: UserId) = ??? val userId: UserId = UserId("some_user_id") val movieId: MovieId = MovieId("some_movie_id") recordView(userId, movieId)
  39. 39. case class MovieId(value: String) extends AnyVal case class UserId(value: String) extends AnyVal def recordView(movieId: MovieId, userId: UserId) = ??? val userId: UserId = UserId("some_user_id") val movieId: MovieId = MovieId("some_movie_id") recordView(userId, movieId)
  40. 40. opaque type MovieId = String opaque type UserId = String def recordView(movieId: MovieId, userId: UserId) = ??? val userId: UserId = "some_user_id" val movieId: MovieId = "some_movie_id" recordView(userId, movieId)
  41. 41. Episode 7: implicit classes SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU 暗黙のクラス
  42. 42. “I learned implicits today. The one that made me smile is implicit classes. You can be so lost if you are not aware of what it is.” — Yannick implicits を学んでわかったこと。 暗黙のクラスは私を笑顔にしてくれる。しかしその正 体に気づかなければ、途方に暮れることでしょう。
  43. 43. import scala.concurrent.duration.Duration import java.util.concurrent.TimeUnit implicit class DurationsOps(n: Long) { def seconds = Duration(n, TimeUnit.SECONDS) } def wait(duration: Duration) = Thread.sleep(duration.toMillis) wait(1.seconds)
  44. 44. import scala.concurrent.duration.Duration import java.util.concurrent.TimeUnit def (n: Long) seconds: Duration = Duration(n, TimeUnit.SECONDS) def wait(duration: Duration) = Thread.sleep(duration.toMillis) wait(1.seconds)
  45. 45. Episode 8: Type class hack SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU 型クラスのハック
  46. 46. trait Encodable { def encode: String } case class Movie(title: String, duration: Int) extends Encodable { override def encode: String = ??? } def send(data: Encodable): Seq[String] = { val raw = data.encode // Send the data }
  47. 47. case class Movie(title: String, duration: Int) extends Encodable { override def encode = ??? }
  48. 48. case class Movie(title: String, duration: Int) extends Encodable with Comparable[Movie] { override def encode = ??? override def compareTo(other: Movie) = ??? }
  49. 49. case class Movie(title: String, duration: Int) extends Encodable with Comparable[Movie] with Cloneable { override def encode = ??? override def compareTo(other: Movie) = ??? }
  50. 50. case class Movie(title: String, duration: Int) extends Encodable with Comparable[Movie] with Cloneable with Serializable { override def encode = ??? override def compareTo(other: Movie) = ??? }
  51. 51. case class Movie(title: String, duration: Int) extends Encodable with Comparable[Movie] with Cloneable with Serializable with Madnessable { override def encode = ??? override def compareTo(other: Movie) = ??? override def becameMad = 🤪 }
  52. 52. Datamanipulations Data Structure case class Movie(title: String, duration: Int) extends Encodable with Comparable[Movie] with Cloneable with Serializable with Madnessable { override def encode = ??? override def compareTo(other: Movie) = ??? override def becameMad = 🤪 }
  53. 53. class Int extends Encodable { override def encode = _.toString }
  54. 54. trait Encoder[T] { def apply(o: T): String } object Encoder { implicit val movieEncoder: Encoder[Movie] = ??? implicit val intEncoder: Encoder[Int] = _.toString } def send[T](data: T) (implicit encoder: Encoder[T]): Seq[String] = { val raw = encoder(data) // Send the data }
  55. 55. def send[T](data: T) (implicit encoder: Encoder[T]): Seq[String] = { val raw = encoder(data) // Send the data }
  56. 56. def send[T](data: T) (implicit encoder: Encoder[T], httpClient: HttpClient): Seq[String] = { val raw = encoder(data) // Send the data }
  57. 57. def send[T](data: T) (implicit encoder: Encoder[T]): Seq[String] = { val raw = encoder(data) // Send the data } send(Movie("Black Mirror", 1000))(0) type mismatch; found : Int(0) required: Encoder
  58. 58. def send[T](data: T) (implicit encoder: Encoder[T]): Seq[String] = { val raw = encoder(data) // Send the data } send(Movie("Black Mirror", 1000)).apply(0)
  59. 59. trait Encoder[T] { def apply(o: T): String } implied for Encoder[Int] = _.toString implied for Encoder[Movie] = ??? def send[T](data: T) given (encoder: Encoder[T]: Seq[String] = { val raw = the[Encoder[T]](data) the[HttpClient].send(raw) // Send the data }
  60. 60. trait Encoder[T] { def apply(o: T): String } implied for Encoder[Int] = _.toString implied for Encoder[Movie] = ??? def send[T](data: T) given (encoder: Encoder[T], client: HttpClient): Seq[String] = { val raw = the[Encoder[T]](data) the[HttpClient].send(raw) }
  61. 61. SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU Ep9: Untyped equality 型チェックされない同値性
  62. 62. case class Movie(title: String, duration: Int) val blackMirror = Movie("Black Mirror", 1000)
  63. 63. def isDuration1Sec(movie: Movie) = movie.duration == 1000
  64. 64. import scala.concurrent.duration._ case class Movie(title: String, duration: Duration) val blackMirror = Movie("Black Mirror", 1.second)
  65. 65. def isDuration1Sec(movie: Movie) = movie.duration == 1000
  66. 66. -language:strictEquality def isDuration1Sec(movie: Movie) = movie.duration == 1000
  67. 67. -language:strictEquality def isDuration1Sec(movie: Movie) = movie.duration == 1.second
  68. 68. Episode 10: Trait parameters SCALA 3, WHAT DOES IT MEANS TO ME? JOAN GOYEAU Traitパラメーター
  69. 69. trait Service[F[_]: ConcurrentEffect] { def fetchSubtitles(movieOffset: Int): F[String] }
  70. 70. NETFLIX PRESENTATION TEMPLATE FOOTER Episode 11: Cross compilation mess クロスコンパイルの複雑さ。Scala 3はすごい。でも、マイナーバージョンアップでさ え苦痛なのに、どうやってScala 3へ移行すればいいのでしょう?
  71. 71. Scala 2.11 Code JVM Bitcode My Project Scala 2.11 Code 2.11 JVM Bitcode Dependency 2.11プロジェクトは、2.11のライブラリに依存します。
  72. 72. Scala 2.12 Code JVM Bitcode My Project ❌ Scala 2.11 Code 2.11 JVM Bitcode Dependency 2.12 へのアップグレードでは、Scala コンパイラが生成するバイトコードが非互換なので、すべて の依存関係を 2.12 でコンパイルし直さなくてはなりません。一方Javaは、JVM をコントロールし ているため、破壊的な変更をJVM側でハンドルできるのです。
  73. 73. Scala 2.12 Code JVM Bitcode My Project Scala 2.12 Code 2.12 JVM Bitcode Dependency 必要なのは、依存関係をアップグレードすることだけです。しかし、 すべての 2.11 ラ イブラリをコンパイルすることは、もはやできません。
  74. 74. Scala 2.12 Code 2.12 JVM Bitcode 2.11 JVM Bitcode JVM Bitcode My Project Scala 2.11/2.12 Code Dependency だから、Scala のバージョンをまたいで、クロスコンパイルします。新たにサポートす るバージョンを追加し、古いバージョンをサポートから外すのです。
  75. 75. Scala 3.0 Code JVM Bitcode My Project 3.0 JVM Bitcode 2.13 JVM Bitcode Scala 2.13/3.0 Code Dependency Scala 3 がリリースされたら、新しい機能を使わずに Scala 3 にクロスコンパイルする 必要があります。
  76. 76. Scala 3.0 Code JVM Bitcode JVM Bitcode Dependency TASTy My Project Scala 2.13 Code
  77. 77. Scala 3.0 Code JVM Bitcode My Project JVM Bitcode Dependency TASTy Scala 2.13 Code JS ECMAScript Native さらに、どんなターゲットに対してもコンパイルできるようになります。この機能は、 Scala 3 への移行がスムーズになるよう, 2.14 に導入されます。
  78. 78. ありがとう dotty.epfl.ch scastie.scala-lang.org jobs.netflix.com

Editor's Notes

  • Hi // today I’m going to speak about Scala 3 / and what it MEANS to us // but BEFORE we start / I’d like tell you a bit MORE about what we do in my team / and what we use Scala for. My name is Joan / and I’m a Data Engineer in the Streaming Data Science & Engineering team at Netflix
  • This friend is basically saying Netflix is just that.
  • We are responsible for 15% of the internet traffic.
  • And it goes all the way to 40% at peak time.
  • As you may know, Netflix is fully hosted on AWS. We don’t own any server.
    And you are using the network of a provider like NTT docomo.
  • If we apply our scale it would probably look like this.
    Both the ISP and Netflix would have to pay a lot in transit cost.
    The ISP wouldn’t be happy as we exploded their network for which they support the full cost.
  • So we fit servers directly into the ISPs infrastructure.
    The ISP and Netflix get back their transit cost
    And the latency is much lower for the users
  • This is the holy grail of Netflix
    Like the CAP theorem:
    If you want to start fast either you cut on quality or you will have more rebuffers.
    If you want high quality either you start slow or you will have more rebuffers.
    If you don’t want rebuffers either you cut on quality or you buffer before starting.
    We believe we can push the boundaries more and more.
  • To do so, every decision at Netflix is Data Driven.
    Anything we do is AB tested.
  • For example, we take like a million users that we try a UI change on and compare with another million on the baseline
    We have many tests in parallel at scale.
  • We’ve been able to improve our encoders to compress a 2h movie to...
    5,924 downloadables
  • We AB even the images and the trailers
  • There is also another type of testing we do at Netflix. When someone tells us that it should failover if we have an issue, we say ok lets try it in prod.
  • So data is core to Netflix from measuring if anything is working well, through testing changes, to business decisions.
    And at Netflix we are using quiet some Scala to process the ton of data we collect.
    We’ve all heard about Scala 3 either through Martin Odersky’s great talks or feature listing on Dotty’s website and we are very excited by this upcoming release.
    But it’s often hard for us, users of Scala, to understand how it will impact our ways of coding and what issues we are trying to solve. So I thought I’ll reverse everything and speak about the annoying stuff in Scala 2 and talk about how Scala 3 aims to tackle these.
  • It’s good because we get those functions to get all values or withName
  • You might wonder, this is all cool but how are we going to go on Scala 3?
    It’s already painful between minor versions.
  • A 2.11 project depends on a 2.11 library.
  • If we upgrade to 2.12 then all our dependencies need to be compiled to 2.12.
    Why? The bitcode generated by the Scala compiler is different, for example in 2.12 we are now compiling higher order functions as java lambdas whereas before it was a hack with classes.
    Why java doesn’t have this issue? Because they control the JVM so they can handle the breaking change on the JVM side.
  • All we need is upgrade the dependencies.
    But now all the 2.11 libraries won’t compile anymore.
  • So we cross compile between Scala versions
    As we add new supported Scala versions we drop the old ones.
  • When Scala 3 will be released libraries would have to cross compile to Scala 3 without using all the new features.
  • Even better you could now compile to any target.
    So this feature will be in 2.14 so that we can have a smooth transition to Scala 3.
  • Arigatō

×