Correctly Architecting your Solutions for Analytical & Operational Uses reviews the two main types of use cases that can be solved with the Denodo Platform. Both high concurrency scenarios and big reporting use cases are discussed in this presentation in a comparative way, explaining the different approaches that you must take to be successful in any situation. This presentation is part of the Fast Data Strategy Conference, and you can watch the video here goo.gl/wdZgpo.

1. Correctly Architecting your Solutions
for Analytical & Operational Uses
Alberto Bengoa, Sr. Product Manager, Denodo

2. Agenda1.Introduction
2.Operational use cases
3.Analytical use cases
4.Deployment architectures
5.Optimization of use cases

3. Introduction

4. 4
Introduction
• The Denodo Platform is a powerful middleware
• Horizontal, rather than vertical
• It can be deployed in a range of use cases
• Each use case is different
• The use case guides the way we deploy data virtualization
• Two main groups of use cases:
• Operational (or transactional)
• Analytical (or informational)

5. Operational Use Cases

6. 6
Operational Use Cases
• Online/interactive work: a human is in an interactive session with
the software
• This requires low latency in the queries
• Queries with low selectivity: each query uses only a small subset
of our data
• Short queries: queries are fast and return small data sets
• Transactional: read & write support with many data sources
• High concurrency: typically there are many clients
Characteristics

7. 7
Operational Use Cases
• Goal: retrieve information about a single customer
• Multiple data sources
• Billing
• Sales
• Technical support
• Potentially write back new information about the customer case
• Many (1000s+) concurrent agents solving customer problems
• Short queries
Example: call center software

8. Analytical Use Cases

9. 9
Analytical Use Cases
• Offline use cases: queries are typically scheduled
• Latency is not important
• Processing of very big data sets
• Long running queries
• Read-only
• Low concurrency
• Infrequent queries
Characteristics

10. 10
Analytical Use Cases
• Create reports across multiple data sources
• Daily, weekly, monthly, quarterly, yearly…
• Very big input data sets
• Fact tables (sales!)
• Multiple dimensions
• Products
• Territories
• Small output data sets
Example: LDW reporting

11. Reference Architectures

12. 12
Denodo Platform Reference Architectures
Operational use cases

13. 13
Denodo Platform Reference Architectures
Analytical use cases

14. 14
Denodo Platform Reference Architectures
Analytical use cases (extended)

15. Optimization and
Deployments

16. 16
Optimization of Use Cases
• Optimize queries for < execution time
• Online use cases have a human waiting for results
• Increase data source pool sizes
• Data source pools can be a bottleneck with many concurrent queries
• Configure high number of concurrent users
• Make sure no clients are waiting unnecessarily
• Hardware resources (CPU, RAM) permitting
• Use historical analysis of queries to characterize workloads
• Know your queries: median and mean duration, number of concurrent queries,
etc.
Operational uses (I)

17. 17
Optimization of Use Cases
• Use partial caching for queries that are very frequent
• Partial caching with low TTL is specially useful during a single agent session
• Scale horizontally: more servers before bigger servers
• Each query is small, so many servers add capacity and resiliency
• Bigger servers do not help with the network bottleneck
• Drop queries vs waiting queues – your choice
• Use case requirements will determine what is best
Operational uses (II)

18. 18
Optimization of Use Cases
• Single sign on, pass-through credentials
• Ensure the data source applies security restrictions when accessing the
data
• Set up low logging levels (logging is slow)
• Make sure logging is done only as absolutely necessary
• Possible to enable higher levels temporarily to diagnose issues
Operational uses (III)

19. 19
Optimization of Use Cases
• Critical to push down processing to the sources
• Analyze the query plans and optimize for delegation
• Push down reduces network data transfers by orders of magnitude
• High memory settings
• Source pushdown is key, but sometimes processing at the data
virtualization layer is needed
• Set up swapping
• Ensure that the server does not run out of memory
• Gracefully degrade performance
Analytical use cases (I)

20. 20
Optimization of Use Cases
• Fast hard drives for swapping
• SSDs will help reduce the impact of accessing primary storage
• Balance query optimization between speed, memory usage and
data source load
• Depending on the limitations of the source systems, the requirements of
the use case, and the hardware of the data virtualization layer
• Set low number of concurrent connections
• Ensure your server is not competing for resources with other queries
Analytical use cases (II)

21. 21
Optimization of Use Cases
• Increase query timeouts
• Standard timeouts are too small for long-running queries
• Monitor server in real time
• Check the health of the servers and ensure everything is ok
• Our monitoring tools allow you to inspect query plans in real time to verify a
correct query behavior
• Use full cache
• To cache slow sources
• To store intermediate results that are reused several times through the report
Analytical use cases (III)

22. 22
Optimization of Use Cases
• Scale vertically: bigger servers before more servers
• Bigger servers allow faster processing of big volumes of data
• More servers do not help in a scenario with few concurrent queries
• Multi-level servers for minimal network transfers and maximum
push down to sources
• Multiple data virtualization servers, close to geographically distributed data
sources
• Each server aggregates local data
• Each server shares with other servers aggregation results (small data sets)
Analytical use cases (IV)

23. 23
Deploying Mixed Solutions
• Use separate servers for separate use cases
• Do not mix use cases in the same server
• Very different performance profiles
• Treat your operational and analytical servers separately
• Assign different hardware resources
• Assign separate security configuration
• Assign separate update policies
• Assign separate code deployment policies

24. Thanks!
www.denodo.com info@denodo.com
© Copyright Denodo Technologies. All rights reserved
Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical,
including photocopying and microfilm, without prior the written authorization from Denodo Technologies.