TIBCO StreamBase: Real-Time Analytics: The Last Mile in Big Data
Richard Tibbetts, CTO, TIBCO StreamBase
ANALYTICS AND VISUALIZATION FOR THE FINANCIAL ENTERPRISE CONFERENCE
June 25, 2013 The Langham Hotel Boston, MA
TIBCO StreamBase: Real-Time Analytics: The Last Mile in Big Data
1Enabling Real Time Analytics in the Capital MarketsRichard TibbettsCTO, TIBCO StreamBaseReal Time Analytics:The Last Mile inBig Data
2Real Time Visibility: The Last Mile in Big DataThe Last Mile:Delivery tousage site• Expensive• Purpose Built• Non-Optional
3Big Data: Everyone Talks about VolumeIn the past 10 years, the amount ofdata to be processed has exploded.Mobile devices, sensors, marketdata, video, audio, and social mediaare among the sources of thisexplosion of data.
5Most solutions focus on accumulating and analyzing
6Especially analyzing things which already happened
7But how will you access the data when you need it?
8Closing the loop – creating value with big data Accumulate – Capture and store the data• Use the right technology to capture data as it happens (e.g. Flume)• Load it into appropriate price/performance storage (e.g. HDFS) Analyze – Form and test hypotheses• Discover mathematical relations and patterns (e.g. R)• Crunch enormous volumes of data to validate and tune (e.g. Hadoop) Access – Use the knowledge at the point of decision• Get the data to the people who need it, when they need it• Present it in the way that they can consume it Automate – Optimize even small decisions with data• Control systems with a closed feedback loop• Monitor and focus on the exceptions, automate the common actions
9Real Time AnalyticsStreamBase Event Processing Platform
10StreamBase EventFlow - Integration, Analytics, Reaction Visual programming integrates components developed by teammembers with different skills• Developers, Quants, Data Scientists, Traders, Business AnalystsConnectivityDataInfrastructureQuantitativeAnalyticsAuditLoggingBusinessStrategyOff the ShelfComponents
11ApplicationsIntegrationNormalizationAggregationCorrelationRulesAlertsAutomationAdaptersandHandlersAdaptersandHandlersStreamBase Server(s)StreamBase Studio forDeveloping EventFlow ApplicationsData ManagementPersistence StoresLogsMarketDataTrade DataStatic DataSystemsDataPerformanceBenchmarksAutomationDesktopAlertsStreamBase Platform for Complex Event ProcessingInputs Outputs
15Kosmos: StreamBase for High Frequency Trading Who is Kosmos?• One of Brazil’s leading hedge fundswhich employs both quantitative andqualitative investment strategies StreamBase for HFT• Connectivity to wide selection ofequities and FX data sources• Building next generation tradingsystem• Equities, FX and derivatives• Selected StreamBase over other CEPproviders Why StreamBase?• Rapid time-to-market• Flexibility• Developer empowerment
16SUNGARD: Pre-Trade Risk Solution Business Driver:• Fast Response to Regulatory Change Project:• Enhancement of Assent Liquidity Solutions andBrass Order Management System Asset classes:• Equities• Derivatives• Fixed income Why StreamBase?• Maturity of StreamBase’s platform and success inthe CEP market• Graphical event-flow language and visualenvironment that enable developers and businessusers to work more efficiently• Superior performance to reduce latency issues“CEP... is very good for rapiddevelopment of applicationswhich are deployed in high-volume, low-latency typeenvironments.The products offered by Assentare constantly being challengedwith lots of market data, ordersand execution updates. Wewant to use StreamBase toaugment and enhance some ofthe products that we have.”Chris Lees, VP Liquidity Services,SunGard
17BNY ConvergEx: StreamBase for Monitoring BNY ConvergEx: The Convergence of Brokerageand Technology• Top 10 NYSE liquidity provider• Vortex, an electronic ATS / dark pool• A leader in algorithmic trading and OMS technology• Deployed StreamBase for monitoring of U.S. tradingoperations BNY ConvergEx Deploys StreamBase for U.S.Trading Operations• Intelligent monitoring of trading operations,including its dark pool Vortex• Real-time market interaction monitoring• Real-time credit risk correlation• System health monitoring• Real-time analysis of market feeds for latency andresponse time“The implementation ofStreamBase allows us tomonitor our risk… andassess the health of ourentire U.S. tradingoperations in real-time.”Joseph Weisbord, ManagingDirector, BNY ConvergExGroup
18CityIndex: StreamBase for Options Pricing and Monitoring Who is CityIndex?• Leading global provider of retail tradingservices, including CFDs, margined FXand spread betting StreamBase for Pricing and Monitoring• Real-time access to prices• Rapid application development anddeployment Why StreamBase?• Rapid time-to-market• High performance• Ease-of-useWatch what Toby Young has to sayabout StreamBase(Requires 2D barcode scanner or click to view)
19InternetHadoopMRMRMRMRMRMRMRMRMRMRMRMRMap ReduceStreamBase CEPMonitoringAutomationFilteringAlertingReal-Time ApplicationsData WarehouseHadoop FlumeMySQLHDFSAnalyticAnalyticAnalyticAnalyticOperational MonitoringLiveViewHistorical AnalyticsAnalystsOperational StaffFraud AnalystsFraud detectionFraud, SecurityManagement ControlSite MetricsBusinessIntelligenceReal Time Big Data: Virtual Goods Phishing and Fraud
20Operational Visibility for Real Time AnalyticsStreamBase LiveView
21Example End User ExperienceReal-Time GraphsActive TablesAd-hoc queryNotificationsEnd-user configurability
22StreamBase LV Use Case - Real-Time Risk ManagementReal-time exposure by asset classReal-timetransaction detailReal-time exposure by desk
23EventFlow and R: Pre-Deal Real Time VAR ChecksNormalizeIntegrate with RSimulationsLoop each tradethrough all scenariosCalculate per-trade,per-scenario RiskCalculate Risk:95%,99%,X%Accept/Rejectproposed trade,minimize risk tocounterpartyAggregateConfigure parametersfor back testingConnect
24StreamBase LiveView Use Case - Real-Time TCAReal-Time TCA AlertsLive Orders, TCA, FIXWhich orders areslipping?
25LiveView Server – Data Source and Query ClientsMessage BusMarket DataNewsOrders &ExecutionsDataLiveView ServerSnapshotResultsContinuous Query ProcessorQueryContinuousAlertingActive Tables Active TablesContinuousUpdatesSources Clients Maintain Data Image Respond to ContinuousQueries Materialize analysis Pre-aggregate data Manage alerts
27GAIN Capital: FX Pricing, Risk, Surveillance and Compliance GAIN Capital• Improved risk management, surveillance and compliancesystem• Rapid applications design, development and deployment• Faster execution GAIN’s developers• Intuitive development environment• Library of application examples and help files• Responsive support team GAIN’s Forex.com clients• Tighter spreads for customers through improved pricing andmarket making StreamBase/GAIN Institutional Clients• Rapid FX applications design, development and deployment• Allow developers to focus on strategies and not the plumbing• High frequency trading connectivity and order execution toall FX liquidity providers including tier 1 and tier 2 banks“StreamBase Event Flowprogramming language is50% faster than traditionalprogramming techniques.”Andrew Haines,CIO, GAIN CapitalWatch what Andrew Haines has to sayabout StreamBase on TABB Forum(Requires 2D barcode scanner or click)
28Summary and Next Steps Once you have accumulated, analyzed, and accessed big datayou will have insight Operationalizing insight on real time data is Event Processing StreamBase’s Event Processing Platform is best of breed• Natural and powerful EventFlow authoring environment• Extensible integrations with analytics• High performance deployment server• Rich portfolio of connectivity with focus on capital markets• LiveView for operational monitoring and alerting Download and try out http://streambase.tibco.comQuestions?