Fpx talk 2014

1,470 views
1,357 views

Published on

Published in: Economy & Finance, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,470
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
22
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • CLICK!
  • This talk is going to be about taking a Java API which offers a very imperative, “close to the metal” mechanism for recovering and persisting data, and then building a functional scala API on topCLICK!
  • OK, so this is my background1 minuteCLICK!
  • There’s not a whole lot to find about GSA out there - essentially we are a technology-driven quant hedge fund and we’ve just won EuroHedge’s award for consistent performance over 5 years in our fund category. Practically every piece of software used in the company, and this amounts to hundreds of systems, we have built ourselves. I work in the CORE technology group, which is a lot more business-focused than it sounds – my “speciality” is on the post-trade execution side of GSA. Particularly reconciliation, allocation and P&L.2 minutesCLICK!
  • So, I’m slightly worried that I have aimed this talk at the wrong audience. From my previous experience, and anyone who’s seen any of my previous talks can attest, I tend to aim at the “don’t know much about any of this” end of things – as I think a lot of people are probably here because they are interested in this stuff. But they can’t all be a RunarBjarnason, Paul Chiusano or Edward Kmett.If you are all like that, then you are going to be very, very bored. Feel free to spend the time griping on twitter.This talk is about taking a Java API for a trade capture system, called furnace. And turning that Java API into a functional scala one, whilst at no point understanding what you are doing.3 minutes
  • Our central repository of trades is called Furnace. It’s a Java system which is, at its most basic view, an NIO server sat atop a SQL database. We bridge trade flow into furnace and we query (and subscribe, though I won’t be going into that here) the furnace event stream via a Java client API.As such, internal real time P&L and position keeping systems can maintain a real time view of GSA’s trading positions, our risks - and realtime reconciliation systems can ensure that our trades are being independently verified, alerting us quickly to any trading issuesFurnace stores events. Each event has an event ID and a transaction ID. Events Ids are unique and monotonically increasing. You can insert new transactions, and an event will be created whose event ID and transaction ID are the same. Subsequent modifications to this transaction take the form of new events, sharing the transaction ID but with a new event ID. You can think of a transaction as being a “Trade” - Each modification to that trade is represented as a subsequent event on the transaction. Events are not deltas, they are the full transaction state at this point in the event sequenceIt’s important to understand that a query will return an event stream – this is typically (although not always) not what you want, however, because you will generally only be interested in the “last” known state of any given transaction. As such, the client API can be a bit clunky to work with.4 minutes
  • Using the furnace Java API is very repetitive:Create a transaction serviceCreate a sessionQuery an event stream (actually more complicated than an iterator)Slice & dice eventsClean up our resources (close sessions etc)I’ve actually simplified the query interface here, because it returns a blocking iterator (as IO is going on under the covers), so this is a resource that also needs to be closedTypically you want some specific things – This query is used *everywhere*. In trade reporting, EOD reconciliation, trade loading into back office systemsObviously we could add some extra layer to the Java Api. But, hey.Also, we probably want to add more idiomatic scala access to properties of the furnace domain model – for example trade.counterparty instead of trade.getCounterparty5 minutes
  • So – let’s start. Dealing with Furnace is IO-like, so let’s create a type called Furnace and make it look a bit like IO. It’s going to wrap a value which was computed during access to the resource.It has a couple of type constructors – the first, which represents the extraction of a value from the interaction (i.e. We can construct an instance of our furnace type given a function which extracts a value from a furnace session). The second wraps any value inside our type6 minutes
  • Anyone familiar with scalaz’s IO in scalaz v6 will know that it will not take long before you overflow the stack. Luckily, some terribly clever people wrote some terribly clever code based on free monads. RunarBjarnason gave a talk about this at scaladays 2012.We gleefully steal his ideas here. In scalaz you’ll see a type called IvoryTower which carries a value thru your computations. We don’t need it here – we can just use furnace sessionWe then modify our Furnace type to contain an apply method that looks like this – essentially, given a furnace service, return the service plus a result in a Trampoline7 minutes
  • We now declare an IO function, taking a function which turns our furnace session into a trampoline. We can then create our type “constructors” for Furnace on top of this, using the return_ construct provided by the trampolineNote that the constructor takes a *function* from furnace session to something! The “point” constructor is trivially a pass thru to our “primary” constructor, which ignores its input and just returns the value you gave it8 minutes
  • Then we implement map and flatMap using furnace and apply. This is all pretty much exactly how IO is done in scalaz and has required minimal thought. I don;t even want to concentrate hugely on what’s going on here because it’s not the point of the talk. The point is: it can be done and it’s not hard.Remember that iogivs us back a furnaceIs that it? Yes.9 minutes
  • Of course, we need a side effect, at least at some point. We need to get a value out of our dealings with Furnace.I’ve done this here in the companion object, rather than the Furnace type itselfWe create a service, construct a session using some (possibly implicitly provided login details), construct our ivory tower from the session, pass this into the furnace type, which will return us a trampoline.We then run the trampoline, which will give us a pair back of our tower and the result – we return the result, finally closing our resourcesAs I said, this is a simplification – in reality things like blocking iterators must be closed as well, but the point stands – all this can be hidden inside the API and the developer shielded from both having to remember to do it, and having it obscure the intent of their code10 minutes
  • We declare an implicit scalaztypeclass for the furnace monad10 minutes
  • Here we start defining, on a package object, some useful methods which will be the nuts & bolts of our API. Note that these useful methods return *functions* from furnace session to something – precisely what our Furnace type constructor took as an argumentThey are very simple to write – they just call methods on the underlying session, scala-fying the result11 minutes
  • Now – we’re going to get back things like Furnace[List[Event]] if we construct a Furnace using our date query, but one of our goals was to have useful things, like “latest” transactions, which were a Map of transactions ids to events (where the latest event was not deleted). Simple – just add more implicit classes!Here I decorate a Furnace of List of events and return a Furnace of a map of events. It will appear to be available, as a method called “latest” wherever we have a Furnace of list of eventsI’m glossing over some more stuff in here, both from my own library (the implies addition to Boolean) and the typesafe “not equals” from scalaz (plus my Equal typeclass generator for a Java enum)12 minutes
  • So imagine we have some method which takes a collection of non deleted transactions and does something with them, maybe it sends an email or creates a file. We can construct a furnace instance using a date query, call “latest” on it to get our map, and then map that to the report invocation, giving us back a Furnace[Unit] (i.e. A side effect). We perform the side effect by calling unsafePerformIO14 minutes
  • OK – so what about inserting stuff? Going back to our DSL package object, we write the insert function11 minutes
  • Now - ifwe call the insert method defined in our package object, then we’re going to get a java future back. And we know there’s not a lot you can do with a Java future, other than block on it and go and make a nice cup of tea CLICK It turns out that if you are in control of the submission of a callable/runnable, it’s rather trivial to return a “listenable” future (i.e. One which you can add/remove listeners on). You can then use this as a basis for constructing a scala future, for which you only really need a call to onComplete. In this code example, FutureListener is my own interface and we are imagining that I’m implementing the onComplete method from within the body of a listenable future. It’s utterly trivial. All the “complication” , such as it is, is in making sure that the addition of a listener is not susceptible to race conditionsNote that the above code invokes f on the listener callback thread – in reality you’d organize this to be done by the execution context16 minutes
  • OK – a fraught conversation with the library author of the furnace Java API has persuaded them to add a ListenableFutureCLICK as a type in their library. <<pause>>With that, and an implicit wrapper, which I’m glossing over here – it gives us the asScala conversion – the insert method no longer contains any trace of Java futures17 minutes
  • If you are not familiar with scalaz, you can probably skip this – essentially we are creating typeclass instances for a scala future. CLICK I’ve overridden ap here, rather than inherit the monad implementation, which is not parallel. This is not without cost, because it will mean that the sequence operation defined later will blow the stack. I’ve just shown you here what it should probably look like!Note that the traversal is implemented via blocking, which is inevitable in the case of sequencing futures. The fact that instances of these typeclasses exist will be a boon later, though!18 minutes
  • OK – we went on a bit of a tangent about scala futures – basically this was so we could remove Java futures from the Scala API. We replaced Java futures with Scala ones and then we added some typeclass instances for them (which the raw scalaz library doesn’t provide). This means we can write something like this...19 minutes
  • OK – here’s a pipeline. What are we doing?We query trades for todayWe make a calculation “non-flat wash positions” – it’s not important what this is, suffice to say that, if this collection is empty, I want to confirm a subcollection of trades from the original query and return this as a success (a right-biased Or). If it’s not empty, I return an exceptionThis constructs a pipeline – note that nothing happens unless I perform the pipeline. Let’s home in a bit on the confirm_ method, because this would not have been possible without our tinkering with futures21 minutes
  • OK – the signature of the confirm_ method is as follows – it’s a Furnace of a Future of a Stream of event ids. How did I implement this method? CLICKWell, I have a stream of furnace events and, for each event, I can insert it using our functional API. Unfortunately CLICK this returns a Stream of Furnace of Futures of event idsBut - we can use “sequence” CLICK to turn a F[G[A]] into an G[F[A]], if F and G have the correct shape. Here, we want to turn a Stream of Furnace into a Furnace of Stream CLICK, possible because Furnace is an applicative functor. But now we have map followed by sequence, which I can just replace by traverseUCLICKNow I’m almost there – I just want to map CLICK the inner Stream of Future into a Future of Stream, which I do using sequence CLICK.There is one problem here – with my applicative future instance defined as it is currently, this inner sequence will blow the stack. We’ll talk about one approach for dealing with this at the end, if I have time, but the second approach I’m going to show you on the next slide. The sclaa library provides a sequence method on the future companion module, so rather than choose the scalaz one, I can choose the scala one instead, which won;t blow the stack23 minutes
  • And we can make this nicer with some more implicit wrappers – where I have a furnace of a stream of futures of something, I can have a furnace of a future of a stream of something. Confirm now looks much nicer.Let’s go back and look at our pipeline again. 24 minutes
  • We query furnace, slice the data, perform many updates, all within the “program” defined by the Furnace type24 minutes
  • So we now concentrate a bit on error handling. If you remember the result type of our pipeline, it encapsulated the possibility of an error case using disjunction. But what of other error cases? What about exceptions resulting from our furnace query? What about exceptions inserting trades? What about exceptions in the intermediate steps of the pipeline?How do we handle those?25 minutes
  • I’m not sure how many of you in the room will be familiar with monad transformers. Our pipeline is a “program” being ran in a computational context, which is a monad. In our case the Furnace monad. But there are other computational contexts that we might wish to run in as well – for example, if you are using the reader monad to “inject” configuration into your program. The problem here is that monads don’t compose. You cannot wrap one monad inside another and still have a monad.So there are these things called monad transformers. In many cases, you can create bespoke transformers for specific monads, where those monads have a particular structure. The resulting transformer is both a monad but also can look a lot like the type it’s transforming. The key understanding here is that, if you deal with *any* monad M to get an M[Banana], and Banana has a transformer, then the result looks a lot like a Banana, when viewed in a particular light.26 minutes
  • Let’s add a new type constructor to our furnace wrapper, except this one will handle errors. Remember the apply takes a function from a session to some type, which we want here to be a disjunction. We can convert this to an eitherT trivially27 minutes
  • Now let us revisit our pipeline – this time threading the computation through EitherT. It looks pretty much the same – it only has a few differences. For exampleWe can only call “latest” on a Furnace of List of Events (w is a List of events)Our confirm method must now return an EitherT but we no longer have the ugly map right and map leftWe have to grab the Furnace of disjunction by calling run on the pipeline29 minutes
  • Let’s implement the confirm method again, but insert exception handling. Note this error handling is going to short circuuit the computation at the first sign of trouble – in reality you might wish to accumulate errors, which can be done, but which I’m not going to show you here.As before, we don’t have much at our disposal, CLICK so we can only really called insert but using eitherT this time. CLICK This is going to give us a Stream of either Ts. Would be nice if we could turn this inside out. Perhaps we could do this as before, via a traverse. CLICK Does that work?Yep – we’ll get an EitherT of furnace/throwable/stream of futures. Remember this is “equivalent” (if you see what I mean) to a Furnace of a disjunction. Remember I said that a bananaT looks a lot like a Banana? Well, the disjunction type has a map method (it’s a functor) and you can map across the RHS of the disjunction. CLICK In our case we wanted to turn the right hand side, a stream of futures, inside out, into a future of a stream CLICK . So we want to map and sequence the inside CLICK 32 minutes
  • So now let’s look at our old confirm method, and our new one. Our old confirm method returned a furnace of confirmed events, by traversing its input with a function to return a furnace of a future and then sequencing its inside.Our new method, which implements error handling, is utterly different. Look, there’s a new full stop and 7 letters in there. And one of those letters is uppercase!So – we just added error handling to our pipeline. Except the pipeline looked the same and the implementation methods look the same. That is precisely the advantage of making a functional API – you get to take advantage of all this stuff that’s been written for you and just works because a) it’s rigorous and b) the people who wrote it are terribly clever (did I mention that?)33 minutes
  • OK – that’s basically the end of this talk. At a high level, we took a simple Java API which was very imperative and we turned it into a functional scala one. We used trampolining to avoid stack overflows. Because GSA had control of the library, we were able to change the source API to provide the ability to listen to futures, on top of which we were able to contruct a scala future. Because scala futures come with their own set of goodies, we could then take advantage of functional libraries like scalaz to generate combinators: chunks of reusable code.That’s the killer here. Why did we do this? So that we could re-use other people’s code. So that I could write stuff quicker. For example, Two pipelines returning Maps whose values are positions – I can just add them, Furnace is a Monoid because a Map is a Monoid because Position is a Semigroup.35 minutes
  • Remember our latest method, which we added to Furnace? We couldn’t call it on an EitherT and so our pipeline needed to have an extra step. We could add new wrappers for lots of transformer types, but it would be really nice is we could abstract over this.Really what we require is just a functor – that is, any container of a list of furnace events can be turned in to a container of a map. The fact we are doing this within the confines of furnace is by the by36 minutes
  • Right – this slide is thanks to Kenji Yoshida and is type wizardry of the most hideous kind. We use a mechanism called “Unapply” (it’s been the U in sequenceU and traverseU on previous slides) to grab a functor. These things here are dependent types – that is, they are type variables of a method parameter.38 minutes
  • And now, let’s look at our pipelines again, the new one on top, the original at the bottom. You can see how similar they are40
  • So – we had a problem with overflowing the stack with sequenceU. We have two strategies for dealing with this. The first is very similar to what we just did with latestF – we decorate any functor with the ability to sequence an inner collection of futures. CLICKThe second choice, which is probably a good idea anyway, is to make it explicit which applicative instance is being imported for future 40
  • Fpx talk 2014

    1. 1. Name Chris Marshall @oxbow_lakes GSA Capital Partners LLP March 2014 Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261
    2. 2. Wrapping an imperative API in a functional one
    3. 3. Mathematics Degree Working in financial software since 1999 • Smalltalk for ~6 months • Java thereafter • Scala since Dec 2008 JP Morgan for 6 years • ~200,000 employees GSA Capital for 7 ¾ years • Quant hedge fund • ~130 employees Background who am i Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 3
    4. 4. Low-latency links & feeds • Up to 800,000 trades /day • 108 market events / day Historic / Current Market Data • Listing changes (e.g. SUNW.O becomes JAVA.O becomes ORCL.O) • News Events Backtesting / Execution Framework Everything Else • This is where I come in • What are our positions? What is our P&L? • Are our trades reconciled? • Reporting (brokers, regulators, administrators) GSA What do we do? Roughly half the company are technologists Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 4
    5. 5. Embarrassingly basic stuff • Target audience (people who’ve not done this before) • Scala programmers working on the JVM Scala • I’m going to assume you can read scala code • I’m going to assume familiarity with typeclasses and their formulation in scalaz • I’m going to assume you understand implicits Overview • Outline of (simplified) imperative API • Wrapping in functional scala • A tangent on futures • An example • Extending the example 5 This talk Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261
    6. 6. Furnace • Furnace is one of GSA’s central systems • It’s a trade capture system which contains all of GSA’s daily trading activity • Currently handles flow of up to 800k trades/day • Plus many more cash events • Including state-transitions (modifications, confirmations etc), it is growing by ~2m events per day • Here’s an extremely simplified view of what its (Java) client API looks like 6 Furnace: an imperative Java data API Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 interface FurnaceService { FurnaceSession createSession(String, String); //user, password } interface FurnaceSession { Iterator<FurnaceEvent> replayEvents(Date); Future<Long> insertEvent(FurnaceEventBuilder); void close(); }
    7. 7. Java/Scala interop • Scala’s Java interop means this is pointless, right? Rationale • furnace’s API provides a very low-level view of interacting with the system • ARM • Idiomatic property access • For the hell of it Typical Query • Give me a Map[TxnId, Event] where the values are the latest event on each transaction, for those transactions whose latest event is not in state DELETED 7 Why implement a wrapper at all? Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261
    8. 8. 8 The Basic Wrapper Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 A datatype “Furnace” • Representing the calculation of a value from the interaction • Obviously, this should be a functor and a monad • We should be able to put an arbitrary value inside our “program” • We need to be able to derive a value from a session sealed trait Furnace[+A] { def unsafePerformIO(): A def map[B](f: A => B): Furnace[B] def flatMap[B](f: A => Furnace[B]): Furnace[B] } object Furnace { def point[A](a: => A): Furnace[A] def apply[A](f: FurnaceSession => A): Furnace[A] }
    9. 9. 9 StackOverflowException Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 Trampolines • Go and look at how scalaz 7 defines IO • Ivory Towers! import scalaz._; import Scalaz._; import Free._ sealed trait Furnace[+A] { ... private def apply(rw: FurnaceSession): Trampoline[(FurnaceSession, A)] }
    10. 10. 10 (I don’t follow this either) Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 Now define some more stuff • A trampolining IO operation • Implement the module “constructors” trait FurnaceFunctions { def furnace[A](f: FurnaceSession => Trampoline[(FurnaceSession, A)]) = new Furnace[A] { private def apply(rw: FurnaceSession) = Suspend(() => f(rw)) } } object Furnace extends FurnaceFunctions { def point[A](a: => A) = apply(_ => a) def apply[A](f: FurnaceSession => A) = furnace(rw => return_(rw -> f(rw))) }
    11. 11. 11 “rw” is Real World Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 Now for map/flatMap • They make use of the previously defined furnace and apply sealed trait Furnace[+A] { ... def map[B](f: A => B) = furnace(rw => apply(rw) map { case (nw, a) => (nw, f(a)) }) def flatMap[B](f: A => Furnace[B]) = furnace(rw => apply(rw) flatMap { case (nw, a) => f(a)(nw) }) }
    12. 12. 12 Getting a value out Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 unsafePerformIO • Resource management object Furnace { ... def unsafePerformIO[A](fa: Furnace[A])(implicit l: Login): A = { val s = new FurnaceService //simplifying val t = s.createSession(l.user, l.password) try fa.apply(t).run._2 finally t.close() } }
    13. 13. 13 Typeclasses Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 object Furnace { module => ... implicit val FurnaceInstances = new Monad[Furnace] { def point[A](a: => A) = module.point(a) def bind[A, B](fa: Furnace[A])(f: A => Furnace[B]) = fa flatMap f } }
    14. 14. 14 That was it? Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 We can now start defining some useful methods • Querying package gsa.furnace package object dsl { def dateQuery(date: Date): FurnaceSession => List[FurnaceEvent] = fs => { import collection.JavaConverters._ fs.replayEvents(date).asScala.toList } }
    15. 15. 15 Useful wrappers Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 Decorate the basics implicit class FurnaceW(f: Furnace[List[FurnaceEvent]]) { def latest(excludeDeleted: Boolean = true): Furnace[Map[Long, FurnaceEvent]] = f map { list => list.foldLeft(Map.empty[Long, FurnaceEvent]) { (m, ev) => m + (ev.getTransactionId -> ev) } filter { case (_, v) => excludeDeleted implies (ev.getState =/= DELETED) } }
    16. 16. 16 Where have we got to? Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 A simple program • Report is cheating a bit – probably should interleave with the IO monad • The expression inside the unsafePerformIO is a Furnace[Unit] def report(evs: Iterable[FurnaceEvent]): Unit = //create a file, send an email? import gsa.furnace.dsl._ val x = Furnace(dateQuery(today)).latest().map(xs => report(xs.values) implicit val login = Login(“cmarsha”, “CorrectBatteryHorseStaple”) Furnace.unsafePerformIO(x)
    17. 17. 17 Updating Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 More useful methods • Inserting package gsa.furnace package object dsl { ... def insert(ev: FurnaceEventBuilder): FurnaceSession => j.u.c.Future[j.l.Long] = fs => fs.insert(ev) }
    18. 18. 18 Still vestiges of Java  Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 We are still working with j.u.c.Future • It’s actually rather trivial to create a “listenable” future • If you control the mechanism of submission (Use CompletionService and inform the listeners on the completion thread) We want to use s.c.Future • Must implement a few methods, isCompleted, onComplete, ready, result, value • Easy to convert a Listenable future into a s.c.Future def onComplete[U](f: Try[T] => U)(implicit e: ExecutionContext) = { val l = new FutureListener[T] { def onCancelled() = try f(Failure(new CancellationException)) finally removeListener(this) def onResult(r: T) = try f(Success(r)) finally removeListener(this) def onThrowable(t: Throwable) = try f(Failure(t)) finally removeListener(this) } addListener(l) }
    19. 19. 19 Our victory is complete! Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 We modify our wrapper • We first persuade the Furnace library author to give us this! • Then we modify our API to remove the last traces of the old order package gsa.furnace package object dsl { ... def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long] = fs => fs.insert(ev).asScala.map(_.longValue) } interface FurnaceSession { ... ListenableFuture<j.l.Long> insertEvent(FurnaceEventBuilder); }
    20. 20. 20 Aside: Scalaz s.c.Future instances Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 implicit def FutureInstances(implicit EC: ExecutionContext) = new Monad[Future] with Traverse[Future] { def point[A](a: => A) = Promise.successful(a).future override def map[A, B](fa: Future[A])(f: A => B) = fa map f override def ap[A, B](fa: => Future[A])(fab: => Future[A => B]) = (fa zip fab) map { case (a, f) => f(a) } //for (f <- fab; a <- fa) yield f(a) def bind[A, B](fa: Future[A])(f: A => Future[B]) = fa flatMap f def traverseImpl[G[_], A, B](fa: Future[A])(f: A => G[B]) (implicit A: Applicative[G]): G[Future[B]] = A.map(f(Await.result(fa, Duration.Inf)))(b => point(b)) }
    21. 21. Where were we again? • We’ve seen how we could query furnace in a more “functional” style • We were removing Java futures, which were returned by the insert method and replacing them with scala futures Why? • Well, we can now compose a “pipeline” which involves querying and updating (inserting) events • What does one of those look like? 21 Quick recap Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261
    22. 22. 22 A “pipeline” where we query/update Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 A look at the finished article type CorrectedEvents = s.c.Future[Stream[Long]] val correctFxRates: Map[Currency, Double] = ??? def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Set[Currency] val pipeline: Furnace[CorrectedEvents] = for { x <- Furnace(dateQuery(d)).latest() //x is Map[Long, FurnaceEvent] y <- suspiciousFxRates(x.values).point[Furnace] z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace] } yield z Await.result(Furnace.unsafePerformIO(pipeline)(login), atMost = 5.minutes)
    23. 23. 23 Let’s home in on correct_ Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 type CorrectedEvents = s.c.Future[Stream[Long]] //recall wrapper package object gives us def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long] def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): Furnace[CorrectedEvents] = xs.toStream.map(e => Furnace(insert(e.withFx(rates(e.ccy))))) //<error>: is a Stream[Furnace[Future[Long]]] = xs.toStream.map(e => Furnace(insert(e.withFx(rates(e.ccy)))).sequenceU //<error>: is a Furnace[Stream[Future[Long]]] = xs.toStream.traverseU(e => Furnace(insert(e.withFx(rates(e.ccy)))) //<error>: is a Furnace[Stream[Future[Long]]] = _.map(_.sequenceU) //is a Furnace[Future[Stream[long]]] is a Furnace[CorrectedEvents]
    24. 24. 24 But we can pretty this last bit up Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 Another implicit wrapper later • We can now implement correct_ as follows implicit class FurnaceFutureStreamW[A](f: Furnace[Stream[Future[A]]]) { def sequenced(implicit EC: ExecutionContext) = f map (Future.sequence(_)) } def correct_(xs: Iterable[FurnaceEvent] ], rates: Currency => Double) = xs.toStream.traverseU(e => Furnace(insert(e.withFx(rates(e.ccy))))).sequenced
    25. 25. 25 A “pipeline” where we query/update Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 val pipeline: Furnace[CorrectedEvents] = for { x <- Furnace(dateQuery(d)).latest() //x is Map[Long, FurnaceEvent] y <- suspiciousFxRates(x.values).point[Furnace] z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace] } yield z
    26. 26. 26 Error Handling Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 Recall the type of our pipeline • We “cheated” by using inserting an exception into our future • What if dealing with furnace throws exceptions? • What if computing our suspicious FX rates throws exceptions? val pipeline: Furnace[CorrectedEvents] def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Exception / Set[Currency]
    27. 27. 27 Monad Transformers: EitherT Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 What’s one of those then? • Monads do not compose (an M[N[A]] is not a monad, even if both M and N are) If Banana is a monad with the correct structure • We denote its transformer as BananaT • Then for any monad M, M[Banana[A]] gives us a BananaT[M, A] • BananaT is a monad • BananaT looks like a Banana There are monad transformers for many monads in Scalaz • StreamT, ListT, OptionT, StateT, ReaderT, EitherT • But not all monads (no FutureT, IOT)
    28. 28. 28 Error Handling Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 Adding error handling to the furnace wrapper • Pretty trivial object Furnace { ... def eitherT[A](fa: FurnaceSession => A): EitherT[Furnace, Throwable, A] = EitherT(apply(fs => /.fromTryCatch(fa(fs)))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Furnace[Throwable / A] }
    29. 29. 29 A “pipeline” handling errors Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 A look at the finished article type CorrectedEvents = s.c.Future[Stream[Long]] def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Throwable / Set[Currency] val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { w <- Furnace.eitherT(dateQuery(d)) x <- EitherT.right(w.point[Furnace].latest()) y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z Furnace.unsafePerformIO(pipeline.run)(login) // Exception / CorrectedEvents
    30. 30. 30 Let’s home in on correct_ (again) Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 type CorrectedEvents = s.c.Future[Stream[Long]] //recall wrapper package object gives us def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long] def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): EitherT[Furnace, Throwable, CorrectedEvents] = xs.toStream.map(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) //<error>: is a Stream[EitherT[Furnace, Throwable, Future[Long]]] = xs.toStream.traverseU(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) //<error>: is a EitherT[Furnace, Throwable, Stream[Future[Long]]] // Furnace[Throwable / Stream[Future[Long]]] = _.map(_.sequenceU) //is a EitherT[Furnace, Throwable, Future[Stream[long]]] //is a EitherT[Furnace, Throwable, CorrectedEvents]
    31. 31. 31 What kind of wizardry is this? Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 //Before def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): Furnace[CorrectedEvents] = xs.toStream .traverseU(e => Furnace(insert(e.withFx(rates(e.ccy))))) .map(_.sequenceU) //After def correct_(xs: Iterable[FurnaceEvent] , rates: Currency => Double): EitherT[Furnace, Throwable, CorrectedEvents] = xs.toStream .traverseU(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) .map(_.sequenceU)
    32. 32. 32 A “pipeline” handling errors Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 A look at the finished article type CorrectedEvents = s.c.Future[Stream[Long]] def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Throwable / Set[Currency] val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { w <- Furnace.eitherT(dateQuery(d)) x <- EitherT.right(w.point[Furnace].latest()) y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z Furnace.unsafePerformIO(pipeline.run)(login) // Exception / CorrectedEvents
    33. 33. What I’ve shown you • We’ve scala-fied a Java API • We’ve removed resource management from a programmers’ concern • We can create blocks of reusable, composable code • We have been able to take advantage of the higher-order abstractions provided by libraries like scalaz What I’ve not shown you • The library throttles the rate at which updates can be made, all transparently to the client • Using other monad transformers with the library (like StreamT) • Using the library with Monoids • Lots more wrappers • Value classes (unboxed typesafe primitives) for transaction id, event id, String properties etc What you might do yourself • Implement ListenableFutureExecutorService which returns listenable futures • Given a listenable future, implement map and flatMap. It’s not as trivial as it might first appear. 33 So what was the point? Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261
    34. 34. 34 Extra – abstracting over “latest” Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 We couldn’t call latest on EitherT • Or OptionT, StreamT etc. • We’d need to declare multiple implicit conversions for each type, wouldn’t we? • Wouldn’t we? implicit class FurnaceW(f: Furnace[List[FurnaceEvent]]) { def latest(excludeDeleted: Boolean = true): Furnace[Map[Long, FurnaceEvent]] = f map { list => ... } }
    35. 35. 35 We wouldn’t Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 We can abstract over a functor of List of events • I’ll see your type-fu and raise you • Thanks, Kenji Yoshida final class ListOfEventFunctorOpsW[F[_]: A](self: F[A], F: Functor[F]) { def latestF(excludeDeleted: Boolean = true) (implicit A: A =:= List[FurnaceEvent]): F[Map[Long, FurnaceEvent]] = F.map(self) { a => A(a) ... } } implicit def listOfEventFunctorOps[FA](fa: FA)(implicit F: Unapply[Functor, FA]) = new ListOfEventFunctorOps[F.M, F.A](F(fa), F.TC) ^^^ ^^^ dependent types
    36. 36. 36 And now, eternity Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { x <- Furnace.eitherT(dateQuery(d)).latestF() y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z x <- Furnace(dateQuery(d)).latest() y <- suspiciousFxRates(x.values).point[Furnace] z <- (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace]
    37. 37. 37 StackOverflows - approaches Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 final class SequencedFunctorOps[F[_], A](self: F[A], F: Functor[F]) { def sequencedF[B, M[_] <: TraversableOnce[_]] (implicit EC: ExecutionContext, A: A => M[Future[B]], cbf: CanBuildFrom[M[Future[B]], B, M[B]]): F[Future[M[B]]] = F.map(self)(a => Future.sequence(A(a))) } object fewchaz { object Implicits { object sequential { implicit def FutureApplicative(implicit EC: ExecutionContext) = { /* do not override ap */ } } object parallel { implicit def FutureApplicative(implicit EC: ExecutionContext) = { /* override ap */ } } } } import fewchaz.Implicits.sequential._
    38. 38. 38 ReaderWriterStateT Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 type R = SmartProps type W = IO[Unit] //or any monoid type S = Unit type F[+A] = Furnace[A] type E = ErrorMsg type M[+A] = ReaderWriterStateT[F, R, W, S, A] type Program[+A]= EitherT[M, E, A] object Program { def furnace[A](f: R => Furnace[E / A]): Program[A] = EitherT[M, E, A](ReaderWriterStateT[F, R, W, S, E / A] { case (r, s) => f(r).map(or => (IO(()), or, s)) } ) def apply[A](f: R => E / A): Program[A] = furnace(f andThen (_.point[F])) def either[A](value: E / A): Program[A] = apply(_ => value) def fromTryCatch[A](value: => A): Program[A] = either(try /-(value) catch { case NonFatal(e) => -/(ErrorMsg(e)) }) def report(msg: String): Program[Unit] = EitherT.right[M, E, Unit](ReaderWriterStateT[F, R, W, S, Unit] { case (r, s) => (IO(println(msg)), (), s) }) }
    39. 39. 39 ...ReaderWriterStateT Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 lazy val program = for { _ <- fromTryCatch(ProductUniverse.setUniverseProvider(new CoreUniverseProvider)) _ <- report(s"About to run the program with: [${args mkString " "}]") dd <- date _ <- report(s"Running for carry date [$dd]") ps <- positions(dd) _ <- report(f"Found ${ps.size}%,d positions in DI1 futures") cc <- //Only try and get marks and D1 if there *are* positions ps.nonEmpty ?? (for { ms <- marks(dd) _ <- report(f"Found ${ms.size}%,d marks on DI1 futures") d1 <- d1Rate(dd) _ <- report(s"D1 rate is $d1") xx <- either(costs(dd, ps.values.toStream, ms)(d1)) } yield xx) _ <- report(s"Costs for positions are: $cc") fs <- //Must run the furnace section regardless of whether we now have costs furnaceInsertOrUpdate(cc, dd) rs <- { import scala.concurrent.duration._ fromTryCatch(Await.result(fs, atMost = 5.minutes)) } _ <- report(f"Created ${rs.size} furnace events") } yield ()
    40. 40. 40 Nothing but a Gnab Gib Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 case class Result(log: W, v: E / Unit) val result = (for { c <- config //E / SmartProps l <- c.maybeLogin(prefix = Some("furnace")).missingConfig //E / Login r <- /-(Furnace.unsafePerformIO(program.run.run(c, ()))(l)) } yield Result(r._1, r._2)) valueOr { s => Result(IO(println(“Unable to invoke program”)), -/(s)) } //The end of the world (result.log |+| IO(println(result.v))).unsafePerformIO
    41. 41. This presentation • http://www.slideshare.net/oxbow_lakes/fpx-talk-2014 • Fewchas: https://gist.github.com/oxbowlakes/8666295 • Furnace Scala API: https://gist.github.com/oxbowlakes/8686567 References • Runar Bjarnason talk: http://skillsmatter.com/podcast/scala/stackless-scala-free-monads • Scalaz: https://github.com/scalaz/scalaz Extras for the masochists Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 41
    42. 42. Contact us Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261 42 GSA Capital Partners LLP investor.relations@gsacapital.com T +44 (0)20 7959 8850 London Office Stratton House 5 Stratton Street London W1J 8LA T +44 (0)20 7959 8800 F +44 (0)20 7959 8845 New York Office 1140 Ave of the Americas 9th Floor New York NY 10036

    ×