ARGUS and Voyanta are now a part of Altus Analytics
BUSINESS NEEDS FOR A
FAST DATA STRATEGY
Gordon Griffin, Chief Architect,
October 2017
ARGUS and Voyanta are now a part of
ABOUT VOYANTA
argussoftware.com | voyanta.com 2
Founded
2012
Launched
2013
Cloud-based
SaaS
Platform
London HQ
New York
Office
Subsidiary of The
Altus Group
Sister company
of ARGUS
Software
ARGUS and Voyanta are now a part of
WHO WE SERVE
3
The World’s Leading Real Estate Companies Use Voyanta
ARGUS and Voyanta are now a part of
WHAT VOYANTA DOES
4
XML
</>
CSV
Abc,
123,
XLSX
Data Quality Filters
Multiple File Formats
???
User
Automation
FTP Upload
File Upload
Input Form
APIs
FTP Export
Data Gathering and
Validation
Data Repository BI and Data
Export
APIs
ARGUS and Voyanta are now a part of
BI DATA MODELLING – CHALLENGES
5
Challenge Original solution
• Broad, complex relational data source
• Transform to numerous complex, tightly specified
reports and dashboards.
• Complex, fine grained permissions model
• Huge cube-type in memory models
• development complexity,
• scaling and performance issues
• Reuse of & access to data model • All in one data model and visualisation solution
• No ability to expose the data model to other uses
(APIs)
• Can’t pull in data from outside the model
• Minimise time from data in to report
availability
• Massive investment in data refresh/reload, auto
scaling infrastructure and queues.
• Customers still waiting over 1 hour for report
refresh.
• Fast, responsive UI & Visualisations:
Minimise time from Click to Presentation.
• In memory cube excels in this area.
ARGUS and Voyanta are now a part of
NEW ARCHITECTURE - VISION
• 3 tier modular architecture:
• Database
• Data Transform
• Visualisation
• Plug and play different data sources, visualisation tech
6
MySQL
MySQL
MySQL
Java
REST
APIs
Data
Transform
BI
Visualisation
Javascript
Excel Add In
ARGUS and Voyanta are now a part of
MODULARITY & REUSE
• Views built on views
• “Facts” can be used as building blocks
• Code reuse
• Consistency
• Compliance
• Separate data transform from
Visualisation
7
Source Tables
Endpoints
(Reports)
Base Views
(facts)
Visualisations
ARGUS and Voyanta are now a part of
DATA TRANSFORMATION
• Create a transformed copy of data
• Require batch builds
• PRO: Rapid queries on pre-built views
• CON: Delay between data in and data
availability
• Mitigate with:
• Incremental builds
• Management of build triggers
• =>Complexity
8
• “ETL On Demand”
• Create Virtual databases, implementing
data transformations
• PRO: Instant availability of data
• CON: complex views on data sources
can be too slow for interactive users
• Mitigate with:
• Partial caching
• Full caching
• => ETL
DATA VIRTUALISATIONETL/CUBE
ARGUS and Voyanta are now a part of
THE “FAST DATA” TRADE-OFF
• Fast UI
• Pre-build ETL or Cube
• Build the answer to all
possible questions
• Hours to build a cube
9
• Fast Data Availability
• Render BI from raw data
• Pull data from the source
without delay
• 30 seconds to render a
dashboard from raw data
VS
ARGUS and Voyanta are now a part of
CACHING IN DATA VIRTUALISATION
• Full Cache
• Built in advance (overnight)
• c.f. ETL
• Data Lag
10
• No Caching
• For simpler/low volume
transformations
• “Time to Live” partial cache
• Simple to implement
• “Cold cache” and cache misses
• Data lag
• “Invalidate and re-query” partial
cache
• Minimise (not eliminate) cold cache
• Requires closer integration
ARGUS and Voyanta are now a part of
“FAST DATA” HEATING
11
Storage Speed
Hadoop,
MapReduce
Cold
In Memory Data
Hot
“Simple” Data
Virtualisation
Partial Caching
Full Caching
Base View ETL
Post DV Extract
Data Delivery
ResponseEmailed
reports
Cold
Customer
Dashboards
Hot
Data
Exploration
Heating requires
• Energy
• Time
ARGUS and Voyanta are now a part of Altus Analytics
BUSINESS NEEDS FOR A
FAST DATA STRATEGYGordon Griffin, Chief Architect,
October 2017

Denodo DataFest 2017: Business Needs for a Fast Data Strategy

  • 1.
    ARGUS and Voyantaare now a part of Altus Analytics BUSINESS NEEDS FOR A FAST DATA STRATEGY Gordon Griffin, Chief Architect, October 2017
  • 2.
    ARGUS and Voyantaare now a part of ABOUT VOYANTA argussoftware.com | voyanta.com 2 Founded 2012 Launched 2013 Cloud-based SaaS Platform London HQ New York Office Subsidiary of The Altus Group Sister company of ARGUS Software
  • 3.
    ARGUS and Voyantaare now a part of WHO WE SERVE 3 The World’s Leading Real Estate Companies Use Voyanta
  • 4.
    ARGUS and Voyantaare now a part of WHAT VOYANTA DOES 4 XML </> CSV Abc, 123, XLSX Data Quality Filters Multiple File Formats ??? User Automation FTP Upload File Upload Input Form APIs FTP Export Data Gathering and Validation Data Repository BI and Data Export APIs
  • 5.
    ARGUS and Voyantaare now a part of BI DATA MODELLING – CHALLENGES 5 Challenge Original solution • Broad, complex relational data source • Transform to numerous complex, tightly specified reports and dashboards. • Complex, fine grained permissions model • Huge cube-type in memory models • development complexity, • scaling and performance issues • Reuse of & access to data model • All in one data model and visualisation solution • No ability to expose the data model to other uses (APIs) • Can’t pull in data from outside the model • Minimise time from data in to report availability • Massive investment in data refresh/reload, auto scaling infrastructure and queues. • Customers still waiting over 1 hour for report refresh. • Fast, responsive UI & Visualisations: Minimise time from Click to Presentation. • In memory cube excels in this area.
  • 6.
    ARGUS and Voyantaare now a part of NEW ARCHITECTURE - VISION • 3 tier modular architecture: • Database • Data Transform • Visualisation • Plug and play different data sources, visualisation tech 6 MySQL MySQL MySQL Java REST APIs Data Transform BI Visualisation Javascript Excel Add In
  • 7.
    ARGUS and Voyantaare now a part of MODULARITY & REUSE • Views built on views • “Facts” can be used as building blocks • Code reuse • Consistency • Compliance • Separate data transform from Visualisation 7 Source Tables Endpoints (Reports) Base Views (facts) Visualisations
  • 8.
    ARGUS and Voyantaare now a part of DATA TRANSFORMATION • Create a transformed copy of data • Require batch builds • PRO: Rapid queries on pre-built views • CON: Delay between data in and data availability • Mitigate with: • Incremental builds • Management of build triggers • =>Complexity 8 • “ETL On Demand” • Create Virtual databases, implementing data transformations • PRO: Instant availability of data • CON: complex views on data sources can be too slow for interactive users • Mitigate with: • Partial caching • Full caching • => ETL DATA VIRTUALISATIONETL/CUBE
  • 9.
    ARGUS and Voyantaare now a part of THE “FAST DATA” TRADE-OFF • Fast UI • Pre-build ETL or Cube • Build the answer to all possible questions • Hours to build a cube 9 • Fast Data Availability • Render BI from raw data • Pull data from the source without delay • 30 seconds to render a dashboard from raw data VS
  • 10.
    ARGUS and Voyantaare now a part of CACHING IN DATA VIRTUALISATION • Full Cache • Built in advance (overnight) • c.f. ETL • Data Lag 10 • No Caching • For simpler/low volume transformations • “Time to Live” partial cache • Simple to implement • “Cold cache” and cache misses • Data lag • “Invalidate and re-query” partial cache • Minimise (not eliminate) cold cache • Requires closer integration
  • 11.
    ARGUS and Voyantaare now a part of “FAST DATA” HEATING 11 Storage Speed Hadoop, MapReduce Cold In Memory Data Hot “Simple” Data Virtualisation Partial Caching Full Caching Base View ETL Post DV Extract Data Delivery ResponseEmailed reports Cold Customer Dashboards Hot Data Exploration Heating requires • Energy • Time
  • 12.
    ARGUS and Voyantaare now a part of Altus Analytics BUSINESS NEEDS FOR A FAST DATA STRATEGYGordon Griffin, Chief Architect, October 2017