Visual Data Discovery with
and Datawatch
Jeremy Bentham
• 28 Aug 2013 –
• Datawatch Completes Acquisition of Panopticon
Datawatch History
• Founded in 1986, Public Since 1992
(NASDQ CM: DWCH)
• Global Operations and Support
 US
 EMEA: UK, G...
What we do?
• Visual Data Discovery
Historically focussed on:
• Front & Mid Office
• Risk, Surveillance, Research, Sales &...
Which Means?
• Reducing the time taken to understand your data.
Effectively:
• Find the Weird Stuff
Using: Designer, Server & Web Client
So From:
To:
Visual Data Display
Time Series
Producing
Competing With
How we’re Differentiated
• Assume data is never at rest
• Capital Markets Focus
• Real Time Streaming
• Time Series
• High...
Kx Connectivity
kx Connectivity
Synchronous: Request / Response
• Issue Q & Retrieve either:
• Table, Dictionary, Vector or Value
Asynchro...
Request / Response Subscribe
kx Connectivity
Kx – How to Query?
Either:
• Retrieve all into Memory
• Parameterise queries, and pull back subsets
• Dynamically query (a...
Problem vs. Competition
Assumed: Data in Motion
So Direct Data Access
• Implying Fast Data Access / Data Querying
So if th...
Solution = Caching
• If data is not time sensitive
• (e.g. Typical data warehouse)
• Populate Cache on a one-off, or sched...
Search for a Cache
We needed an in-memory cache that could:
• Load quickly
• Perform fast aggregation
• Perform fast filte...
Dataset Characteristics
• Typically Sparse Timeseries
• Sensor Data
• Sales/Revenue Transactions
• Latency Data
• Machine ...
Way Forward
• Approached kx for OEM
• But our pricing ruled out usage within the Designer
Then:
• 2nd April – 32bit kx – F...
Next Datawatch Release – Cache Options
• Designer – 32bit kx.
• View Single Workbook at a time
• Server –32bit or 64bit kx...
Our Data Strategy
• If Fast underlying database.
• Go Direct
• If Slowwwwww
• Cache into kx,
• Get the query performance t...
More Information
Peter Simpson
Visual Data Discovery
peter_simpson@datawatch.com
TEL: +44 (0) 798 464 6544
AquaQ Analytics Kx Event - Datawatch Presentation
AquaQ Analytics Kx Event - Datawatch Presentation
Upcoming SlideShare
Loading in …5
×

AquaQ Analytics Kx Event - Datawatch Presentation

1,239 views

Published on

Published in: Software, Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,239
On SlideShare
0
From Embeds
0
Number of Embeds
668
Actions
Shares
0
Downloads
12
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

AquaQ Analytics Kx Event - Datawatch Presentation

  1. 1. Visual Data Discovery with and Datawatch
  2. 2. Jeremy Bentham
  3. 3. • 28 Aug 2013 – • Datawatch Completes Acquisition of Panopticon
  4. 4. Datawatch History • Founded in 1986, Public Since 1992 (NASDQ CM: DWCH) • Global Operations and Support  US  EMEA: UK, Germany, France, Sweden  Asia Pac: Australia, Singapore, Hong Kong, India, Philippines • Pioneer in Transforming All Types of Information  Structured (RDBMs, Data Warehouses)  Semi-Structured (PDF, Reports, Text …)  Unstructured (Log Files, EDI …) • Over 40,000 customers worldwide  99 of the Fortune 100 & 487 of the Fortune 500  Large Number of SMB  Across All Verticals
  5. 5. What we do? • Visual Data Discovery Historically focussed on: • Front & Mid Office • Risk, Surveillance, Research, Sales & Trading • For Buy & Sell Side, Regulators Exchanges & ECNs Now Still Capital Markets plus: • Energy & Utilities, Telco, Retail, Manufacturing, etc.
  6. 6. Which Means? • Reducing the time taken to understand your data. Effectively: • Find the Weird Stuff
  7. 7. Using: Designer, Server & Web Client
  8. 8. So From:
  9. 9. To:
  10. 10. Visual Data Display
  11. 11. Time Series
  12. 12. Producing
  13. 13. Competing With
  14. 14. How we’re Differentiated • Assume data is never at rest • Capital Markets Focus • Real Time Streaming • Time Series • High Density Visuals • Embed (Java & .NET SDKs) • Java & .NET Servers • Connectivity
  15. 15. Kx Connectivity
  16. 16. kx Connectivity Synchronous: Request / Response • Issue Q & Retrieve either: • Table, Dictionary, Vector or Value Asynchronous Subscribe • Subscribe to Service, Table & Symbols • Keeping latest, or scrolling time window
  17. 17. Request / Response Subscribe kx Connectivity
  18. 18. Kx – How to Query? Either: • Retrieve all into Memory • Parameterise queries, and pull back subsets • Dynamically query (auto-generating q selects) Retrieve: • Summaries & Detail • Sampled Time series • Down to individual Ticks Passing through: • Parameter Values & Vectors of Values • Time Windows • Zoom Bounds
  19. 19. Problem vs. Competition Assumed: Data in Motion So Direct Data Access • Implying Fast Data Access / Data Querying So if the underlying data source is: Slow We appear: Slow
  20. 20. Solution = Caching • If data is not time sensitive • (e.g. Typical data warehouse) • Populate Cache on a one-off, or scheduled basis. • Dynamically Querying of Cache • Approach taken by: • Tableau, Tibco Spotfire & Qlikview • Their In-Memory Db = Proprietary Cache
  21. 21. Search for a Cache We needed an in-memory cache that could: • Load quickly • Perform fast aggregation • Perform fast filtering • Work with big datasets • Understand Time • Small footprint • Easy to OEM • Windows & Linux
  22. 22. Dataset Characteristics • Typically Sparse Timeseries • Sensor Data • Sales/Revenue Transactions • Latency Data • Machine Data • Market Data & Trade Data (Orders & Executions) • Everywhere we look across verticals, data seems similar to trades & quotes
  23. 23. Way Forward • Approached kx for OEM • But our pricing ruled out usage within the Designer Then: • 2nd April – 32bit kx – Free for Commercial Use
  24. 24. Next Datawatch Release – Cache Options • Designer – 32bit kx. • View Single Workbook at a time • Server –32bit or 64bit kx Cores • Host Multiple Workbooks • Cache up to the memory in the machine (if using 64bit cores)
  25. 25. Our Data Strategy • If Fast underlying database. • Go Direct • If Slowwwwww • Cache into kx, • Get the query performance that kx provides
  26. 26. More Information Peter Simpson Visual Data Discovery peter_simpson@datawatch.com TEL: +44 (0) 798 464 6544

×