Optiq is a framework that allows heterogeneous data from different locations and formats to be queried using a standard SQL interface. It helps optimize queries by caching results, pre-computing aggregates, and using column-oriented storage and materialized views to speed up analytics queries on large datasets. By treating different data sources as a single database and helping "cheat" through query optimization techniques, Optiq provides the benefits of a centralized database for managing and querying hybrid transactional and analytical workloads.

1. Optiq
A dynamic data management framework
@julianhyde
Friday, October 4, 13

2. 3 things
1. Databases are good.
2. It is hard to build analytics if your data is
“all over the place.”
3. Optiq makes heterogeneous data look and
behave more like a database.
Friday, October 4, 13

3. Mondrian on Optiq on
hybrid data + memory
Friday, October 4, 13

4. Big Data
Friday, October 4, 13

5. Friday, October 4, 13

6. “Data all over the
place”
• Different locations (HDFS, memory, DBMS)
• Different formats
• Different workloads
• Transactional: Mainly write, targeted read
• Analytic: Mainly read (bulk write)
• Query latency: Interactive vs. batch
• Data latency: Can we show out-of-date data?
Friday, October 4, 13

7. Databases are good
• Central point to manage data
• Simple, standard API for apps
• Powerful modeling techniques (e.g. star
schemas)
• Data independence (i.e. tune your data
after you write your application)
• Query optimization
Friday, October 4, 13

8. Optiq
Friday, October 4, 13

9. Conventional DB architecture
Friday, October 4, 13

10. Optiq architecture
Friday, October 4, 13

11. Examples
Friday, October 4, 13

12. Example #1: CSV
• Uses CSV adapter (optiq-csv)
• Demo using sqlline
• Easy to run this for yourself:
$ git clone https://github.com/julianhyde/optiq-csv
$ cd optiq-csv
$ mvn install
$ ./sqlline
Friday, October 4, 13

13. More adapters
Adapters Embedded Planned
CSV Cascading (Lingual) HBase (Phoenix)
JDBC Apache Drill Spark
MongoDB Cassandra
Splunk Mondrian
linq4j
Friday, October 4, 13

14. Example #2:
Splunk + MySQL
SELECT p.product_name,
COUNT(*) AS c
FROM splunk.splunk AS s
JOIN mysql.products AS p
ON s.product_id = p.product_id
WHERE s.action = 'purchase'
GROUP BY p.product_name
ORDER BY c DESC
Friday, October 4, 13

15. Expression tree
Friday, October 4, 13

16. Optimized tree
Friday, October 4, 13

17. Analytics on
heterogeneous data
Friday, October 4, 13

18. Simple analytics
problem
• 100M U.S. census records
• 1KB each record, 100GB total
• 4 SATA3 disks, total 1.2GB/s
• How to count all records in under 5s?
Friday, October 4, 13

19. Simple analytics
problem
• 100M U.S. census records
• 1KB each record, 100GB total
• 4 SATA3 disks, total 1.2GB/s
• How to count all records in under 5s?
Friday, October 4, 13

20. Simple analytics
problem
• 100M U.S. census records
• 1KB each record, 100GB total
• 4 SATA3 disks, total 1.2GB/s
• How to count all records in under 5s?
• Not possible?! It takes 80s just to read the
data.
Friday, October 4, 13

21. Solution: Cheat!
Friday, October 4, 13

22. Solution: Cheat!
• Compress data
• Column-oriented storage
• Store data in sorted order
• Put data in memory
• Cache previous query results
• Pre-compute (materialize) aggregates
Friday, October 4, 13

23. How Optiq helps you
to cheat
Friday, October 4, 13

24. How Optiq helps you
to cheat
• Materialized views
• Pre-defined aggregate tables
• Cached query results = In-memory tables
• Smart cache maintenance
• Quickly bring materializations online & offline
• Materializations over a subset of the data
• Spark distributed, in-memory processing & cache
• Application thinks it is talking to a single SQL database
Friday, October 4, 13

25. Mondrian on Optiq on
hybrid data + memory
Friday, October 4, 13

26. Summary
Friday, October 4, 13

27. 3 things (reprise)
1. Databases are good. Especially the
flexibility that SQL gives us.
2. It is hard to build analytics if your
data is all over the place. Different
workloads (operational vs. analytic, small write
vs. bulk read) require different data structures.
3. Optiq is not a database. But Optiq creates
a federated data architecture that performs
well, and looks like a database to your tools.
Friday, October 4, 13

28. @julianhyde
optiq https://github.com/julianhyde/optiq
mondrian http://mondrian.pentaho.com
blog http://julianhyde.blogspot.com
Friday, October 4, 13