Climbing mount complexity, one event at a time

3,968 views

Published on

How do financial trading systems, with multiple liquidity markets and continuous real-time updates of pricing models, make trade execution decisions to achieve maximum return? In this talk David Vincent explores how open source software, particularly ESPER Complex Event Processing and QuickFixJ are used today in leading Liquidity Management Systems (LMS). The presentation will focus on the architecture of Smart Trade LMS and how open source software is used in real world, high-performance, low-latency applications.

Find a podcast of David's talk at:
http://bit.ly/7HEB1l

Published in: Technology, Business
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
3,968
On SlideShare
0
From Embeds
0
Number of Embeds
1,807
Actions
Shares
0
Downloads
0
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Climbing mount complexity, one event at a time

  1. 1. CLIMBING MOUNT COMPLEXITY, ONE EVENT AT A TIME David VINCENT – CTO © 2009 smartTrade Inc.
  2. 2. Presentation Agenda 1. Liquidity Management Playground 2. Liquidity Management Foundations 3. Implementation challenges 4. QuickFix/J and SmartTrade LMS 5. ESPER and SmartTrade LMS © 2009 smartTrade Inc © 2009 smartTrade Inc.
  3. 3. Liquidity Management Play Ground © 2099 smartTrade Inc © 2009 smartTrade Inc.
  4. 4. Flows in presence © 2009 smartTrade Inc.
  5. 5. Various objectives • Best execution • Best services to clients ( market impact, orders management ) • Algo • Market • Venues • Cross-Asset © 2009 smartTrade Inc.
  6. 6. Business and Economical Dimension • Business • Pricing / Rules / Decision making • Automated Market-Making ( active / passive orders ) to clients and venues • Creating Dark Pools • Pre-matching, Best Exec (RegNMS, MIFID,...) • Access to Liquidity / Connectivity • High Velocity flows • Economical • Market impact • Execution and clearing fees control • More Value/Less Risk/ More Flows © 2009 smartTrade Inc.
  7. 7. Functional Dimension • Supporting various market discovery modes • Streaming • Request For Stream • Supporting various trading modes • Orders • RFQ • Multi Leg patterns • Basis trading • Synthetic products © 2009 smartTrade Inc.
  8. 8. Technical Dimension • Moving targets – always • Complex interacting processes • Old legacy systems part of the game • Low latency • Scalability • High Availability – zero error acceptance • Integration – openness, flexibility, interoperability, acceptability by IT teams © 2009 smartTrade Inc.
  9. 9. Liquidity Management Foundations © 2099 smartTrade Inc © 2009 smartTrade Inc.
  10. 10. Foundation of Liquidity Management Liquidity Aggregator Liquidity Crosser Liquidity Orchestrator Liquidity Connect All components can be used individually, or combined, to help solve entire Liquidity Management needs. © 2009 smartTrade Inc.
  11. 11. The Internal Market Paradigm • Virtualization of all trading flows at the enterprise level: complete internal market state model • Full lifecycle management of all inbound and outbound instructions and orders • Complex distribution and liquidity management strategies (orders types, price aggregation, smart routing, internal crossing, internalization, multi-leg strategies, price-tiering) • Easy-to-add new products and automated rules • Seamless integration to current & existing infrastructures for a short time-to-market implementation (including transport) • Connectivity: Liquidity Connect © 2009 smartTrade Inc.
  12. 12. Generic Liquidity Management System © 2009 smartTrade Inc.
  13. 13. Implementation challenges © 2099 smartTrade Inc © 2009 smartTrade Inc.
  14. 14. Key Technology Factors • Common Data Model • Explicit messages & structures of data shared across the platform are defined in a domain specific XML file data model • Extensive Cross-platform code generation ( Java, .NET, C++ ) • Extensibility – static and dynamic data model fields & structures • Open Messaging • Decouples messaging middleware dependencies from internal architecture • Allows several messaging transports to connect to the same container • Leveraging client’s existing deployments • Dependencies Injection • Offers a easy way to create assemblies of components; allows very flexible configurations • Deployment Supervision • Mix between Agent based approach (non-intrusive) & JMX • Component configuration ; Supervision & Lifecycle Management • Web based application allowing monitoring of: - Hardware state ( CPU, Processes, Disks, Memory ) - Application state ( Threads, Heap ) - Custom Metrics via JMX plug-ins framework © 2009 smartTrade Inc.
  15. 15. Development Approach • SCRUM Methodology • Agile approach to software development & program management, enabling fast and iterative deliveries to clients. • A key principle of Scrum is its recognition that during a project the customers can change their minds about what they want and need, so its gives valuable flexibility to these unpredicted challenges. • XP (Extreme Programming) • A disciplined approach to software development. • XP empowers developers to confidently respond to changing customer requirements, even late in the life cycle. • It stresses customer satisfaction. So projects will experience greater success and developer productivity. • High Involvement in Open Source World • Open Source development philosophy internally • Contributor in multiple OpenSource projects • Project lead of QuickFixJ OpenSource Java FIX engine © 2009 smartTrade Inc.
  16. 16. QuickFix/J : a pure Java FIX Engine © 2099 smartTrade Inc © 2009 smartTrade Inc.
  17. 17. What is FIX ? • Financial Information eXchange • Created in 1992. • Free, open technical specification for electronic communication of trade-related messages. • Maintained by FIX Protocol Ltd. • Volunteer team • In wide use (exchanges, banks, …) • Global adoption © 2009 smartTrade Inc.
  18. 18. FIX protocol layers • Session Protocol • Message format, header, footer • Logon/Logoff • Heartbeats • Sequence number synchronization • Application Protocol • Message formats and semantics • Order submittal and execution • Quoting • Market data • Other (email, news, …) • Representation • ASCII Tag/Value • FIXML (XML, Tag/Value envelope) • Example Message: • 8=FIX.4.19=6135=A49=INVMGR56=BRKR34=152=20000426- 12:05:0698=0108=3010=157 © 2009 smartTrade Inc.
  19. 19. What is a FIX Engine ? • Session protocol implementation • Messages and behavior • Supports application messages • Behavior is application-specific • Eases trading counterparty connectivity • “The nice thing about standards…” • Many variations © 2009 smartTrade Inc.
  20. 20. QuickFix/J Open Source Project • Spin off of the QuickFIX C++ project • Native C++ • Java via JNI • QuickFix/J motivators • Interesting domain • Integration/deployment issues w/JNI • Free FIX engine for Java clients • No existing OSS Java FIX engines • Unhappy with existing vendors ( Price, Support ) • Unhappy with “homebrew” engines • History • Discussed on QF lists and wiki • November 2004 : QFJ work started in by Steve BATE • Help from Smart Trade on the port • Mid-July 2005 : 1.0.0 beta released • December 2008 : Smart Trade Technologies takes over the project lead • February 2009 : 1.4 released • http://www.quickfixj.org © 2009 smartTrade Inc.
  21. 21. QuickFix/J Elements • FIX metadata (XML) • Parser, Pretty Printing, Validation • Generation of Message-related code • Uses Java, XSLT, and XML metadata • Java message and field definitions (FIX 4.0 – 5.0, Serializable ) • Message Cracker (type safe dispatcher) • Application • Interface implemented by users • Several callbacks (Session creation, Logon/Logoff, Message sent/recvd). • Session • Implements FIX protocol • Data dictionary • Message Factory • Message Store (file, JDBC, BDB) • Session Log (file, JDBC, console) • Settings (file) © 2009 smartTrade Inc.
  22. 22. QuickFix/J Elements • Connection management • Socket Initiator (usually buy side) • Socket Acceptor (usually sell side) • Single or multithreaded • NIO using MINA • Alternate implementations possible • Acceptance Testing tools • declarative test definitions derived from FIX spec, • JUnit based • Unit testing tools • fills coverage gaps in acceptance testing • JUnit based • Cruise Control • Continous integration support • Builds product and executes tests • Example applications and code samples © 2009 smartTrade Inc.
  23. 23. QuickFix/J Benefits • Covers the full FIX connectivity scope • Supports FIX versions 4.0 - 4.4, 5.0/FIXT1.1 • Scheduling of session connections • Failover and High Availability • Open Source • Free! It costs nothing and has a very liberal open source license. • Full source code available (also at no cost). • Native Java • Runs on any hardware and operating system supported by 1.4+ Java SE or compatible VM. • Supports embedded SSL with Java 5+ • Provides standard JMX MBeans for FIX engine management • Easy to embed in existing Java applications • Highly Customizable • Support for protocol customizations (new messages, fields, constraints). • Session state storage plugins: JDBC, File, SleepyCat/JE, In memory • Logging plugins: JDBC, File, SFL4J (supports JDK1.4 logging, Log4J, Commons Logging), Console, Composite © 2009 smartTrade Inc.
  24. 24. QuickFix/J inside the Smart Trade LMS © 2099 smartTrade Inc © 2009 smartTrade Inc.
  25. 25. Liquidity Connect Architecture • Connecting N world to P world • Enterprise Integration Patterns implementation • Not yet CAML ☺ • Router / Translator / Interceptor Chains • Connector • Abstract the API to connect to a specific world • Manages life cycle of the connection ( ready, down, stale.. ) • Receives and transmits message to an interceptor chain • Interceptor chain • Unidirectional • Pass messages along different components • Connector is always at the start and the end of a chain • One to many relationship : 1 connector -> N connector • Bridge • Assembly of connectors and interceptor chains • Spring based definition © 2009 smartTrade Inc.
  26. 26. Liquidity Connect Architecture • A –B Bridge • A – BC Bridge © 2009 smartTrade Inc.
  27. 27. STTP Connector • Connecting Smart Trade LMS world to P world • Using the Smart Trade STConnect API • Normalizing data into the Smart Trade canonical model • Translator interceptors © 2009 smartTrade Inc.
  28. 28. QuickFIX/J Connector • Connecting Smart Trade LMS world to FIX world • Using QuickFix/J Fix engine and libraries • Normalizing data into the FIX canonical model • Translator interceptors to manage versions • Support Custom fields © 2009 smartTrade Inc.
  29. 29. QuickFix/J and ST LMS © 2009 smartTrade Inc.
  30. 30. ESPER CEP © 2099 smartTrade Inc © 2009 smartTrade Inc.
  31. 31. Esper Open Source Project • Mission • CEP a natural and easy task of your application • Open platform to create competitive differentiators and maximize reuse • ...and therefore • Lightweight and embeddable into any Java technology process • Open source • Low latency and high throughput • Project background • 3 years of development, 100% Java • NEsper for .NET/C#, same semantics • Dual licensing Enterprise/OEM/Open Source model • Open source for open source use (GPL license) • Commercial for redistribution by ISVs • Services & support (dev, 8x5, 24x7) • http://esper.codehaus.org © 2009 smartTrade Inc.
  32. 32. Esper Products Offering • ESPER • ESP/CEP with expressive Event Processing Language • Continuous queries, aggregation, joins, causality and missing events, joins to historical data, output flow control... • Standalone or Embedded • ESPER HA • Out-of-heap storage, complete resiliency to disk, or local in-memory caching • Clustering and hot backup capabilities for event and streams state • ESPER ENTERPRISE EDITION • Esper Studio: Operational console for runtime management, real- time event stream views and ad-hoc queries • EsperJMX: Runtime management and monitoring over standard JMX connectors • EsperJDBC: JDBC compliant client and server endpoints for interoperability • NESPER • ESPER for C#/.NET © 2009 smartTrade Inc.
  33. 33. Esper Key Benefits • ESPER Integrated ESP and CEP and expressive EPL language • Rapid development, test and reuse of existing code • Java libs, other open source fwk, Eclipse IDE etc. • Performant & lightweight • Server-less if needed • Possible use of Real-Time JVMs • Multi-platform (Java / .Net, 32/64 bits, multiple OS support) • Ease of integration in existing platforms or standards • Spring Framework • Enterprise Integration Technologies • Management console, runtime management, standard based (JMX) © 2009 smartTrade Inc.
  34. 34. ESPER inside the Smart Trade LMS © 2099 smartTrade Inc © 2009 smartTrade Inc.
  35. 35. Why integrating with ESPER CEP ? • CEP are NOT Liquidity Management Systems…… They are complementary • Smart Trade • “Smart Trade’s customers will be able to rely on a domain-specific CEP for their SOR, crossing and aggregation strategies on top of already existing programmable rules that our liquidity management system produces and implements to ensure best execution for their firm and clients. This clearly proves that CEP and liquidity management are complementary technologies.” David VINCENT – co-founder and CTO • Esper Tech • "The EsperTech and SmartTrade agreement combines and extends our open-source roots to maximize fit for purpose and confidence while minimizing costs for Reg NMS and MiFID best-execution deployments. This once again confirms how our enterprise-ready and open-source CEP makes sense in today’s world across all industries.” Thomas BERNHARDT, founder and CEO • OEM Agreement between Smart Trade Technologies and Esper Tech signed in February 2009 © 2009 smartTrade Inc.
  36. 36. ESPER Plugin • Domain specific integration • Creation of complex stream on the fly • Manipulating a Domain Specific model • Wrapping the ESPER 3.2 runtime into STContainers • Rely on EPServiceProvider, EPStatement and the other friends • Using Spring • Extending the ESPER engine with specific behaviour ( ex : sorting ) • Wrapping the Smart Trade canonical model with Event • Ex : MarketDataSnapshot => MarketDataSnapshotEvent • Ex : NewOrderSingle => NewOrderSingleEvent • Using model extensibility to add EPL return statement fields © 2009 smartTrade Inc.
  37. 37. Liquidity Aggregator’s ESPER Plugin • Integration with the STConnect API • Allows to interact remotely with the ESPER Engine • Creating and consuming EPL queries from external process • Chosing feed sources • “Plumbing” Liquidity sources events to the ESPER engine • Source of data • Distribution End point © 2009 smartTrade Inc.
  38. 38. Examples of ST / EPL use • Average Bid in the last 30 seconds window select avg(price.value) as PriceAverage from MarketDataSnapshotEvent(MDReqID.value=‘XXXX')[bidEntries].win:time(30 sec) • Instant BestBid / BestOffer spread select bestAggregatedOffer.price.value-bestAggregatedBid.price.value as Spread from MarketDataSnapshotEvent(MDReqID.value=‘XXXX') • Average BestBid / BestOffer spread in the last 30 seconds window select avg(bestAggregatedOffer.price.value-bestAggregatedBid.price.value) as Spread from MarketDataSnapshotEvent(MDReqID.value=‘XXXX').win:time(30 sec) • VWAP select average as VWAP from MarketDataSnapshotEvent(MDReqID.value=‘XXXX')[bidEntries].win:stat:weighted_avg(price.value, quantity.value) © 2009 smartTrade Inc.
  39. 39. Liquidity Orchestrator’s ESPER Plugin • Generating complex events to be managed by the rules • From aggregated or raw market data • From order life cycle events ( Fills, Cancelation, unsollicitated messages ) • From external functional events ( Risk exposure,..) to technical events ( Latency, ..) • Separating the events monitoring logic from the actions • A clean Action / Reaction model • A way to layer the rules coding © 2009 smartTrade Inc.
  40. 40. ESPER and ST LMS © 2009 smartTrade Inc.
  41. 41. Questions © 2009 smartTrade Inc.
  42. 42. Please contact Europe United Kingdom Tel : + 33 1 44 50 19 19 Tel.: +44 (0)20 3326 0522 Fax: + 33 4 42 97 62 52 Fax: +44 (0)20 7491 7976 hgozlan@smart-trade.net njames@smart-trade.net North America Tel.: +1 212 618 6383 Fax: +1 212 618 6309 mbailey@smart-trade.net www.smart-trade.net

×