Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Scala Corrections Library


Published on

A number of examples why Scala is in my rear view mirror, and a few ideas on how to improve the collections.

Published in: Technology, Education
  • @undefined

    Are you sure you want to  Yes  No
    Your message goes here
  • videos are not present at either of those links --or overlooked since miscategorized. Can you update where these videos would be or confirm they wont be available online?
    Are you sure you want to  Yes  No
    Your message goes here
  • Here are the sites where (as I understand things) the videos will appear: for the one given at linkedin and for the one given at kabam.
    Are you sure you want to  Yes  No
    Your message goes here
  • There will be two videos (I gave the talk twice) and the slides are not written to be understood without explanation so I hope it wasn't too grave a mistake to put them online before the videos were available. I'll post links here once they're online.
    Are you sure you want to  Yes  No
    Your message goes here
  • Interesting! Is a video recording of the talk available?
    Are you sure you want to  Yes  No
    Your message goes here

A Scala Corrections Library

  1. 1. A Scala Corrections Library Paul Phillips Source: xkcd, of course.
  2. 2. “When I'm working on a problem, I never think about beauty. I think only how to solve the problem.” ! “But when I have finished, if the solution is not beautiful, I know it is wrong.” – R. Buckminster Fuller (syntax highlighting donated by paulp)
  3. 3. “When I'm working on a problem, I never think about beauty. I think only how to solve the problem.” ! “But when I have finished, if the solution is not beautiful, I know it is wrong.” – R. Buckminster Fuller trait ParSeqViewLike[ +T, +Coll <: Parallel, +CollSeq, +This <: ParSeqView[T, Coll, CollSeq] with ParSeqViewLike[T, Coll, CollSeq, This, ThisSeq], +ThisSeq <: SeqView[T, CollSeq] with SeqViewLike[T, CollSeq, ThisSeq] ] extends GenSeqView[T, Coll] with GenSeqViewLike[T, Coll, This] with ParIterableView[T, Coll, CollSeq] with ParIterableViewLike[T, Coll, CollSeq, This, ThisSeq] with ParSeq[T] with ParSeqLike[T, This, ThisSeq]
  4. 4. The Winding Stairway • Five years on scala • Rooting for scala/typesafe • But I quit a dream job... • ...because I lost faith
  5. 5. Credentials
  6. 6. Credentials, cont.
  7. 7. Should you care? • I offer my credentials only to bear witness to my credibility • I suspect I have written more scala code than anyone else, ever. • What’s visible in compiler/library represents only a small fraction of it
  8. 8. Caveats • I ran out of time. Slides are rushed. Forgive me. • Error messages and repl transcripts have been heavily trimmed for clarity on a slide • This works counter to message when the point involves complexity or incomprehensibility • So verbosify all compiler messages by a factor of three for a more accurate feel
  9. 9. My axe is dull • I have been pulling my punches • This has left some thinking that I quit over technical esoterica: java compatibility, jvm limitations, intractable compiler challenges • This is not accurate
  10. 10. Subtext, people • Prevailing programmer culture frowns upon criticism of named individuals • In this case that doesn’t leave much room for additional specificity • All the relevant facts are available in the googles
  11. 11. Is Scala too complex? • I’ll field this one: YES • Is anyone fooled by specious comparisons of language grammar size? Who cares? • Half the time when someone hits a bug they can’t tell whether it is a bug in scala or the expected behavior • That definitely includes me
  12. 12. Perceived Problem C • A meme is going around that scala is too complex • Option A: Own it • Option B: Address it • Option C: Obscure it p O i t n o
  13. 13. Thus is born the “use case” // A fictional idealized version of the genuine method def map[B](f: (A) B): Map[B] ! // The laughably labeled "full" signature def map[B, That](f: ((A, B)) B) (implicit bf: CanBuildFrom[Map[A, B], B, That]): That neither has any basis in reality!
  14. 14. the true name of map // markers to distinguish Map's class type parameters scala> class K ; class V defined class K, V ! scala> val host = typeOf[Map[K, V]] host: Type = Map[K,V] ! scala> val method = host member TermName("map") method: Symbol = method map ! // Correct signature for map has FOUR distinct identifiers scala> method defStringSeenAs (host memberType method) res0: String = def map[B, That](f: ((K, V)) => B) (implicit bf: CBF[Map[K,V],B,That]): That
  15. 15. • Now you’re thinking “use case thing is a bug, big deal, bugs get fixed.” Do they? • Surely as soon as it is known the documentation spins these fabrications, it will be addressed? If not fixed, at least it’ll be marked as inaccurate? Something? • Nope! To this day it’s the same. Your time is worthless.
  16. 16. Slightly Caricatured map def map[B](f: A => B): F[B] Signature Elegance Advantages Spokespicture “map” def map[B, That](f: A => B) (implicit bf: CanBuildFrom[Repr, B, That]): That Among the purest and most reusable <—- Not this. abstractions known to computing science Can reason abstractly about code Can map a BitSet to a BitSet without typing “toBitSet”
  17. 17. The Bitset Gimmick // Fancy, we get a Bitset back! scala> BitSet(1, 2, 3) map (_.toString.toInt) res0: BitSet = BitSet(1, 2, 3) ! // Except… scala> BitSet(1, 2, 3) map (_.toString) map (_.toInt) res1: SortedSet[Int] = TreeSet(1, 2, 3) ! // Um… scala> (BitSet(1, 2, 3) map identity)(1) <console>:21: error: type mismatch; found : Int(1) required: scala.collection.generic.CanBuildFrom[scala.collection.imm utable.BitSet,Int,?] (BitSet(1, 2, 3) map identity)(1) ^
  18. 18. similarly scala> def f[T](x: T) = (x, new Object) f: [T](x: T)(T, Object) ! scala> SortedSet(1 to 10: _*) res0: SortedSet[Int] = TreeSet(1, 2, 3, ! scala> SortedSet(1 to 10: _*) map (x => res1: SortedSet[Int] = TreeSet(1, 2, 3, ! scala> SortedSet(1 to 10: _*) map f map res2: Set[Int] = Set(5, 10, 1, 6, 9, 2, 4, 5, 6, 7, 8, 9, 10) f(x)._1) 4, 5, 6, 7, 8, 9, 10) (_._1) 7, 3, 8, 4)
  19. 19. and in a similar vein scala> val f: Int => Int = _ % 3 f: Int => Int = <function1> ! scala> val g: Int => Int = _ => System.nanoTime % 1000000 toInt g: Int => Int = <function1> ! scala> Set(3, 6, 9) map f map g res0: Set[Int] = Set(633000) ! scala> Set(3, 6, 9) map (f andThen g) res1: Set[Int] = Set(305000, 307000, 308000)
  20. 20. Java Interop: the cruelest joke • It’s impossible to call scala’s map from java! • See all the grotesque details at SI-4389 IX F T ON “I played with it until it got too tedious. I think the signatures work fine. What does not work is that the variances of CanBuildFrom cannot be modelled in Java, so types do not match. And it seems Java does not even let me override with a cast. So short answer: You can't call these things from Java because instead of declaration side variance you have only a broken wildcard system.” ! — Martin Odersky W
  21. 21. Lightning Round • My time is running out and I can hear you saying… • “Just give us a laundry list of collections issues” • Okay, you asked for it (in my mind)
  22. 22. • Implementation details infest everything • And every detail is implementation-defined • Capabilities should be designed around the laws of variance; instead variance checks are suppressed and key method contains is untyped • Specificity rules render contravariance useless • Implicit selection and type inference inextricably bound - so type inference is largely frozen because any change will break existing code
  23. 23. • Extreme pollution of base objects - all collections have “size: Int”, all Seqs have “apply”, etc. • Bundling of concerns (e.g. invariant Set) • Inheritance of implementation is the hammer for every nail… • …yet “final” and “private”, critical for a hope of correctness under inheritance, are almost unknown • Semantics discovered instead of designed
  24. 24. assume the worst In Set(x) ++ Set(x), which x wins? ! Can xs filter (_ => true) return xs? ! Are defaults preserved across operations? Which operations? Is sortedness? Will views and Streams retain laziness when zipped?
  25. 25. xs map identity scala> val m = Map(1 -> 2) withDefaultValue 10 m: Map[Int,Int] = Map(1 -> 2) ! scala> m(1000) res0: Int = 10 ! scala> (m map identity)(1000) <console>:9: error: type mismatch; found : Int(1000) required: CanBuildFrom[Map[Int,Int],(Int, Int),?] (m map identity)(1000) ^ ! scala> m map identity apply 1000 java.util.NoSuchElementException: key not found: 1000 at MapLike$class.default(MapLike.scala:228)
  26. 26. types are for suckers % find collection -name ‘*.scala’ | xargs egrep asInstanceOf | wc -l 556
  27. 27. How could 556 casts ever go wrong scala> val xs: Set[Int] = (1 to 3) => x)(breakOut) ! java.lang.ClassCastException: SeqViewLike$$anon$3 cannot be cast to immutable.Set
  28. 28. get and apply trivially fall into disagreement ! scala> Map[Int,Int]() withDefaultValue 123 res0: Map[Int,Int] = Map() ! scala> res0 contains 55 res1: Boolean = false ! scala> res0 get 55 res2: Option[Int] = None ! scala> res0 apply 55 res3: Int = 123
  29. 29. Why is covariance such an object of worship? Types exist so we don’t have to live like this! // WHY infer this utterly useless type? scala> List(1, 2) ::: List(3, 4.0) res0: List[AnyVal] = List(1, 2, 3.0, 4.0) ! scala> PspList(1, 2) ::: PspList(3, 4.0) <console>:23: error: type mismatch; found : PspList[Int] required: PspList[Double]
  30. 30. Type Inference + Variance ——————————
  31. 31. Abstracting over mutability • • • An inherited implementation is ALWAYS wrong somewhere!! • Half the overrides in collections exist to stave off the incorrectness which looms above. This is nuts.! • Not to mention “Map”, “Set”, etc. in three namespaces Example: how do you write "drop" so it's reusable?! In a mutable class, drop MUST NOT share, but in an immutable class, drop MUST share!
  32. 32. How many ways are there to write ‘slice’ ? % ack --no-filename 'def slice(' src/library/ ! 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 override def slice(from: Int, until: Int): Iterator[A] = def slice(from: Int, until: Int): Iterator[A] = { def slice(from: Int, until: Int): Repr = def slice(from: Int, until: Int): Repr = { def slice(from: Int, until: Int): Repr = { def slice(start: Int): PagedSeq[T] = slice(start, UndeterminedEnd) def slice(unc_from: Int, unc_until: Int): Repr override /*IterableLike*/ def slice(from: Int, until: Int): Vector[A] = override /*TraversableLike*/ def slice(from: Int, until: Int): Repr = { override def slice(_start: Int, _end: Int): PagedSeq[T] = { override def slice(from1: Int, until1: Int): IterableSplitter[T] = override def slice(from1: Int, until1: Int): SeqSplitter[T] = override def slice(from: Int, until: Int) = { override def slice(from: Int, until: Int) = { override def slice(from: Int, until: Int): List[A] = { override def slice(from: Int, until: Int): Repr = self.slice(from, until) override def slice(from: Int, until: Int): Repr = { override def slice(from: Int, until: Int): Stream[A] = { override def slice(from: Int, until: Int): String = { override def slice(from: Int, until: Int): This = override def slice(from: Int, until: Int): This = override def slice(from: Int, until: Int): Traversable[A] override def slice(from: Int, until: Int): WrappedString = { override def slice(unc_from: Int, unc_until: Int): Repr = {
  33. 33. scala.conflation • Every collection must have size • Every sequence must have apply • Every call to map includes a "builder factory" • Every set must be invariant • Everything must suffer universal equality
  34. 34. predictability One of these expressions returns 2 and one returns never. Feeling lucky? ! scala> (Stream from 1) zip (Stream from 1) map { case (x, y) => x + y } head ! scala> (Stream from 1, Stream from 1).zipped map (_ + _) head
  35. 35. sets Two complementary ways to define Set[A]. Complementary - and NOT the same thing! Intensional Extensional Specification Membership test Members Variance Set[-A] Set[+A] Defining Signature A => Boolean Iterable[A] Size Unknowable Known Duplicates(*) Meaningless Disallowed
  36. 36. What's going on here? scala> class xs[A] extends Set[A] error: class xs has 4 unimplemented members. ! // Intensional/extensional, conflated. // Any possibility of variance eliminated. def iterator: Iterator[A] def contains(elem: A): Boolean // What are these doing in the interface? // Why can I define a Seq without them? def -(elem: A): Set[A] def +(elem: A): Set[A]
  37. 37. todo: also add all other methods % git grep 'todo: also add' 607cb4250d SynchronizedMap.scala: // !!! todo: also add all other methods ! % git grep 'todo: also add' origin/master SynchronizedMap.scala: // !!! todo: also add all other methods ! commit 607cb4250d Author: Martin Odersky <> Date: Mon May 25 15:18:48 2009 (4 years, 8 months ago) ! added SynchronizedMap; changed Set.put to Set.add, implemented LinkedHashMap/Set more efficiently.
  38. 38. tyranny of the interface • Mandating "def size: Int" for all collections is the fast track to Glacialville! • Countless times have I fixed xs.size != 0 • Collections are both worlds: all performance/ termination trap, no exploiting of size information! • A universal size method must be SAFE and CHEAP
  39. 39. Psp Collections • So here is a little of what I would do differently • I realized since agreeing to this talk that I may have to go cold turkey to escape scala’s orbit. It’s just too frustrating to use. • Which means this may never go anywhere • But you can have whatever gets done
  40. 40. Conceptual Integrity trait Collections { type CC[+X] type Min[+X] type Opt[+X] type CCPair[+X] type ~>[-V1, +V2] ! ! } type type type type type type type type // // // // // Iso[A] Map[-A, +B] FlatMap[-A, +B] Grouped[A, DD[X]] Fold[-A, +R] Flatten[A] Build[A] Pure[A] the overarching container type (in scala: any covariant collection, e.g. List, Vector) least type constructor which can be reconstituted to CC[X] (scala: GenTraversableOnce) the container type for optional results (in scala: Option) some representation of a divided CC[A] (at simplest, (CC[A], CC[A])) some means of composing operations (at simplest, Function1) = = = = = = = = CC[A] ~> CC[A] CC[A] ~> CC[B] CC[A] ~> Min[B] CC[A] ~> CC[DD[A]] CC[A] ~> R CC[Min[A]] ~> CC[A] Min[A] ~> CC[A] A ~> CC[A] trait Relations[A] { type MapTo[+B] = Map[A, B] type FoldTo[+R] = Fold[A, R] type This = CC[A] type Twosome = CCPair[A] type Self = Iso[A] type Select = FoldTo[A] type Find = FoldTo[Opt[A]] type Split = FoldTo[Twosome] } // // // // // // // // // // // // // // // // e.g. filter, take, drop, reverse, etc. e.g. map, collect e.g. flatMap e.g. sliding e.g. fold, but also subsumes all operations on CC[A] e.g. flatten for use in e.g. sliding, flatMap we may not need an alias incorporating the known A another one the CC[A] under consideration a (CC[A], CC[A]) representation a.k.a. CC[A] => CC[A], e.g. tail, filter, reverse a.k.a. CC[A] => A, e.g. head, reduce, max a.k.a. CC[A] => Opt[A], e.g. find a.k.a. CC[A] => (CC[A], CC[A]), e.g. partition, span
  41. 41. “Do not multiply entities unnecessarily” • mutable / immutable • Seq / Set / Map • parallel / sequential • view / regular 24 Combinations!
  42. 42. Surface Area Reduced 96% • A Set is a Seq without duplicates. • A Map is a Set paired with a function K => V. • A mutable collection has nothing useful in common with an immutable collection. Write your own mutable collections. • If we can’t get sequential collections right, we have no hope of parallel collections. Write your own parallel collections. • “Views” should be how it always works.
  43. 43. predictability: size matters scala> def f(xs: Iterable[Int]) = xs.size f: (xs: Seq[Int])Int ! // O(1) scala> f(Set(1)) res0: Int = 1 ! // O(n) scala> f(List(1)) res1: Int = 1 ! // O(NOES) scala> f(Stream continually 1) <ctrl-C>
  44. 44. Asking the right question SizeInfo / Atomic / Infinite Precise Bounded
  45. 45. Don’t ask unanswerable questions (Unless you enjoy hearing lies) scala> val xs = Foreach from BigInt(1) xs: psp.core.Foreach[BigInt] = unfold(1)(<function1>) ! scala> xs.size <console>:22: error: value size is not a member of psp.core.Foreach[BigInt] xs.size ^ ! scala> xs.sizeInfo res0: psp.core.SizeInfo = <inf>
  46. 46. the joy of the invariant leaf scala> List(1, 2, 3) contains "1" res0: Boolean = false ! scala> PspList(1, 2, 3) contains "1" <console>:23: error: type mismatch; found : String("1") required: Int PspList(1, 2, 3) contains "1" ^
  47. 47. HEY! MAP NEED NOT BE RUINED! scala> "abc" map (_.toInt.toChar) res1: String = abc ! scala> "abc" map (_.toInt) map (_.toChar) res2: IndexedSeq[Char] = Vector(a, b, c) ! // psp to the rescue scala> "abc".m map (_.toInt) map (_.toChar) res3: psp.core.View[String,Char] = view of abc ! scala> res3.force res4: String = abc