SlideShare a Scribd company logo
In search of database nirvana
The challenges of delivering Hybrid Transactional and Analytical Processing
Rohit Jain, CTO
rohit.jain@esgyn.com
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Agenda
The swinging database pendulum
Hybrid Transaction and Analytical Processing (HTAP) Workloads
Query versus storage engines
The challenges of HTAP
◦ Single query engine for all workloads
◦ Supporting multiple storage engines
◦ Same data model for all workloads
◦ Enterprise-caliber capabilities
Conclusion
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
The swinging database pendulum
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
RDBMS NoSQL
• TCO
• Elastic scalability
• High performance
• Semi-structured & unstructured data
• Parallelization of user code
• Schema flexibility
• Modest needs
Polyglot programming & persistence
• graph database
• document stores
• text search
• column stores
• key value stores
• wide column stores
• Too many languages, interfaces, APIs,
& data structures
• Too much of gluing technologies together
• Compatibility between different versions
• No end-to-end view of workload performance
• Support contracts with multiple vendors
• Too many skills required to develop and manage
• Too much data movement
• No single solution for varied interfaces & use cases
SQL
• Skills prevalent
• Existing tools & applications
• Transaction support useful
• More efficient when joins needed
• Easier than coding M/R
• Merit in rigor of pre-defining columns
• Uniform metadata across applications
Hybrid Transaction and Analytical
Processing (HTAP) Workloads
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
OLTP
• Mostly transactional
• Sub-second response
• Customer experience
• Large update volume
• High concurrency
• Scales linearly
• Normalized data model
• Custom applications or
3rd party solutions
• Mostly SMP; MPP for
web-scale
• Keyed updates/queries
ODS
• Can be transactional
• Sub-second to seconds
• Customer experience or
Business internal
• Batch to streaming feeds
from OLTP
• Low update volume
• Low concurrency if
internal, high otherwise
• Near linear scale
• Historical data
• Normalized data model
• Custom apps / 3rd party
• Keyed queries
BI
• Non-transactional
• Seconds to minutes
• Business internal
• Batch to streaming feeds
from OLTP/ODS
• No direct updates
• Low to high concurrency
• Less linear in scale
• Historical data
• Dimension data model
• BI tools – reporting &
dashboards
• Ad hoc & scheduled
queries and large extracts
Analytics
• Non-transactional
• Minutes to hours
• Business internal
• Batch/aggregates from BI
• No direct updates
• Low concurrency
• Complex queries, non-
linear scale
• Historical & big data
• Columnar store
• Analytics in database
• Analytical tools
• Ad hoc queries
Essential to operate the business To improve performance of the company
Query versus storage engines
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Hadoop Cluster
Switch Switch
Operational Business Intelligence Analytics
Query Engine
• Allow clients to connect & submit queries
• Distribute connections across cluster
• Compile query
• Execute query
• Return results of query to client
Storage Engine
• Storage structure
• Partitioning
• Automatic data repartitioning
• Select columns
• Select rows based on predicates
• Caching writes and reads
• Clustering by key
• Fast access paths or filtering
• Transactional support
• Replication
• Compression & Encryption
• Mixed workload support
• Bulk data ingest/extract
• Indexing
• Colocation or node locality
• Data Governance
• Security
• Disaster recovery
• Backup, Archive, Restore
• Multi-temperate data support
In-memory
Single Query Engine
The challenges of HTAP
Single query engine for all workloads
Data structure – key support, clustering, partitioning
Statistics
Predicates on non-leading or non-key columns
Indexes and materialized views
Degree of parallelism
Reducing the search space
Join type
Data flow and access
Mixed Workload
Feature support
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
80 minutes 2 minutes
Equal-height
histograms
The challenges of HTAP
Single query engine for all workloads
Data structure – key support, clustering, partitioning
Statistics
Predicates on non-leading or non-key columns
Indexes and materialized views
Degree of parallelism
Reducing the search space
Join type
Data flow and access
Mixed Workload
Feature support
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Week Item Store …
01/07/2016 1 1 …
01/07/2016 1 3 …
01/07/2016 1 5 …
01/07/2016 2 34 …
01/07/2016 3 13 …
01/07/2016 3 3 …
01/07/2016 4 2 …
01/07/2016 4 4 …
01/14/2016 1 2 …
01/14/2016 1 4 …
01/14/2016 1 5 …
01/14/2016 1 35 …
01/14/2016 3 1 …
01/14/2016 3 20 …
Where is item = 1, Stores 2 through 5?
The challenges of HTAP
Single query engine for all workloads
Data structure – key support, clustering, partitioning
Statistics
Predicates on non-leading or non-key columns
Indexes and materialized views
Degree of parallelism
Reducing the search space
Join type
Data flow and access
Mixed Workload
Feature support
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Serial vs parallel plans
Node 1 Node 2 Node n
Client Application
HDFS
HBase
Region 1
Filters
HDFS HDFS HDFS HDFS
Ethernet
Coprocessors
HBase
Region 2
HBase
Region 3
HBase
Region 4
HBase
Region 5
Master Master
Multi-
fragment
Master
ESP ESP ESP ESP ESP
ESP ESP ESP ESP ESP
The challenges of HTAP
Single query engine for all workloads
Data structure – key support, clustering, partitioning
Statistics
Predicates on non-leading or non-key columns
Indexes and materialized views
Degree of parallelism
Reducing the search space
Join type
Data flow and access
Mixed Workload
Feature support
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Qry1
Qry2Qry4
Qry3Qry5 Qry6
Qry7
The challenges of HTAP
Single query engine for all workloads
Data structure – key support, clustering, partitioning
Statistics
Predicates on non-leading or non-key columns
Indexes and materialized views
Degree of parallelism
Reducing the search space
Join type
Data flow and access
Mixed Workload
Feature support
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Adaptive and parallel joins
• Nested join
• Probe cache for nested join
• Merge join
• Matching partition join
• Repartitioned hash join
• Replication by broadcast hash join
• Inner / outer child broadcast
• Dimensional schema star join
• Inner join
• Left Join
• Right Join
• Full Outer Join
• Self join
Cost Premiums for nested joins or
serial plans
The challenges of HTAP
Single query engine for all workloads
Data structure – key support, clustering, partitioning
Statistics
Predicates on non-leading or non-key columns
Indexes and materialized views
Degree of parallelism
Reducing the search space
Join type
Data flow and access
Mixed Workload
Feature support
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Compute
Cost
Execution
Environment
Physical
Properties
Estimates
Confidence
Cardinality,
Distribution,
Correlation
Sensitivity
To Estimates
Evaluate
Risk
Risk
Adjustment
Benefit
Risk
Risk Premiums
• Nested join 20%
• Merge join 10%
• Serial plan 5%


?
Data structure – key support, clustering, partitioning
Statistics
Predicates on non-leading or non-key columns
Indexes and materialized views
Degree of parallelism
Reducing the search space
Join type
Data flow and access
Mixed Workload
Feature support
• Priority / SLA based execution
• Allocation of resources by service level
• Decrease priority with usage increase
• Anti-starvation / switch between
queries based on priority
The challenges of HTAP
Single query engine for all workloads
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Query
Low
Query
Medium
Queue
Memstore
HBase
….
Memstore
HBase
Memstore
HBase
Queue Queue
HBase
Region 1
HBase
Region 3
HBase
Region 5
Query
High
Low Low Low
Medium MediumMedium
High HighHighLow Low Low
Medium MediumMedium
High HighHigh
The challenges of HTAP
Supporting multiple storage engines
Statistics
Key structure
Partitioning
Data type support
Projection and selection
Extensibility
Security enforcement
Transaction Management
Metadata support
Performance, scale, and
concurrency considerations
Error handling
Other operational aspects
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Single-Master Multiple-Masters
The challenges of HTAP
Same data model for all workloads
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Normal form
• 1NF
• 2NF
• 3NF
• BCNF
• 4NF
• 5NF
• 6NF
Star Schema
Snowflake Schema
Normal Form
Query engine integration with storage
engine(s) to support all these data models
The challenges of HTAP
Same data model for all workloads
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
NoSQL Data Models
“NoSQL Data Modeling Techniques”
by Ilya Katsov
Highly Scalable Blog
… and these!
The challenges of HTAP
Enterprise-caliber capabilities
High Availability
Security
Manageability
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
• Percentage of uptime 99.99% = 52.56 minutes
downtime to 99.999% = 5.26
• Online operations (data available for reads and writes)
o Upgrading the OS
o Upgrading the file system
o Upgrading the storage engine
o Upgrading the query engine
o Redistribute data to accommodate node and/or disk
expansions and contractions
o Changing table definition, e.g. data type changes,
and adding, dropping, renaming columns
o Create/drop secondary indexes
o Full and Incremental Backups
The challenges of HTAP
Enterprise-caliber capabilities
High Availability
Security
Manageability
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Schema Management Performance Management Monitoring Security Management BAR Management
Object Management Performance Monitoring Database Monitor User Management Backup Analysis
Graphical Object Editor Live Performance Monitoring Event Monitoring Role Management Recovery
Cross-Platform Schema Knowledge Data Repository Live Event Monitoring Account Migration Log Backup
Bottleneck Analysis Threshold Alerts Audit Report Backup Reports
SQL Management Job/Workload Analysis Health Index Alarm Archival
Query Builder Job/Workload Wizard Live Health Monitoring
Visual Difference Tool Job/Workload Management Response Times Maintenance Configuration Management
Data Management Live Job/Workload
Monitoring
Alert Center Repository Aging OS Provisioning
Data Migration OS Analysis Remote Monitoring Automated Maintenance Cluster Provisioning
SQL Profiler Capacity Capture Central Monitoring Instance Provisioning
Automated Import Capacity Trending Hardware Inventory Change Management Cloud Provisioning
Visual Explain Plans Capacity Forecast Hardware Monitoring Schema Capture Configuration Editor
Session Management Space Management Schema Compare and Synch
Lock Management Reorganization Management Troubleshooting Notifications
Process Management Query Cost Simulation Health Analysis Schema Rotation
Consistency Checks Historical Reports Problem Correlation Collaboration
Online Schema Evolution Bottleneck Tuning Automated Actions Virtual Changes
Built-In Automation Access Path Analysis
The challenges of HTAP
Enterprise-caliber capabilities
High Availability
Security
Manageability
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
• Operational performance by transactions per second
• Analytical performance by query
• Overhead of gathering metrics on operational and analytical workloads
• Configurable statistics collection
• Workload management by Service Level Objectives
o Based on priority and/or resource allocation
o High priority operational workloads vs analytical workloads
• End-to-end visibility of transaction and query metrics
• Metric breakdown down to the query operation
• Metrics for table access across workloads down to the partition level
• Skew or bottlenecks
• Integration with YARN
Conclusion
(C) Copyright 2015 Esgyn Corporation Esgyn Confidential
Pre-register for full O’Reilly report:
http://www.oreilly.com/go/dbnirvana
It ain’t easy!!
Very few products can even come close
Any guesses? 

More Related Content

What's hot

What's new in apache hive
What's new in apache hive What's new in apache hive
What's new in apache hive
DataWorks Summit
 
Presto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performancePresto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performance
DataWorks Summit
 
HTAP Queries
HTAP QueriesHTAP Queries
HTAP Queries
Atif Shaikh
 
What's new in SQL on Hadoop and Beyond
What's new in SQL on Hadoop and BeyondWhat's new in SQL on Hadoop and Beyond
What's new in SQL on Hadoop and Beyond
DataWorks Summit/Hadoop Summit
 
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache Tez
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache TezYahoo - Moving beyond running 100% of Apache Pig jobs on Apache Tez
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache Tez
DataWorks Summit
 
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotReal-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
 
Migrating Oracle to PostgreSQL
Migrating Oracle to PostgreSQLMigrating Oracle to PostgreSQL
Migrating Oracle to PostgreSQL
Amazon Web Services
 
Membase Meetup 2010
Membase Meetup 2010Membase Meetup 2010
Membase Meetup 2010
Membase
 
Apache Spark Based Reliable Data Ingestion in Datalake with Gagan Agrawal
Apache Spark Based Reliable Data Ingestion in Datalake with Gagan AgrawalApache Spark Based Reliable Data Ingestion in Datalake with Gagan Agrawal
Apache Spark Based Reliable Data Ingestion in Datalake with Gagan Agrawal
Databricks
 
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
Michael Stack
 
Leveraging docker for hadoop build automation and big data stack provisioning
Leveraging docker for hadoop build automation and big data stack provisioningLeveraging docker for hadoop build automation and big data stack provisioning
Leveraging docker for hadoop build automation and big data stack provisioning
Evans Ye
 
The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)
The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)
The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)
Ontico
 
Reshape Data Lake (as of 2020.07)
Reshape Data Lake (as of 2020.07)Reshape Data Lake (as of 2020.07)
Reshape Data Lake (as of 2020.07)
Eric Sun
 
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
Data Con LA
 
Graphene – Microsoft SCOPE on Tez
Graphene – Microsoft SCOPE on Tez Graphene – Microsoft SCOPE on Tez
Graphene – Microsoft SCOPE on Tez
DataWorks Summit
 
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
Data Con LA
 
ETL Practices for Better or Worse
ETL Practices for Better or WorseETL Practices for Better or Worse
ETL Practices for Better or Worse
Eric Sun
 
Loading Data into Redshift: Data Analytics Week at the SF Loft
Loading Data into Redshift: Data Analytics Week at the SF LoftLoading Data into Redshift: Data Analytics Week at the SF Loft
Loading Data into Redshift: Data Analytics Week at the SF Loft
Amazon Web Services
 
Real time fraud detection at 1+M scale on hadoop stack
Real time fraud detection at 1+M scale on hadoop stackReal time fraud detection at 1+M scale on hadoop stack
Real time fraud detection at 1+M scale on hadoop stack
DataWorks Summit/Hadoop Summit
 
From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...
From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...
From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...
DataWorks Summit
 

What's hot (20)

What's new in apache hive
What's new in apache hive What's new in apache hive
What's new in apache hive
 
Presto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performancePresto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performance
 
HTAP Queries
HTAP QueriesHTAP Queries
HTAP Queries
 
What's new in SQL on Hadoop and Beyond
What's new in SQL on Hadoop and BeyondWhat's new in SQL on Hadoop and Beyond
What's new in SQL on Hadoop and Beyond
 
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache Tez
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache TezYahoo - Moving beyond running 100% of Apache Pig jobs on Apache Tez
Yahoo - Moving beyond running 100% of Apache Pig jobs on Apache Tez
 
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotReal-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache Pinot
 
Migrating Oracle to PostgreSQL
Migrating Oracle to PostgreSQLMigrating Oracle to PostgreSQL
Migrating Oracle to PostgreSQL
 
Membase Meetup 2010
Membase Meetup 2010Membase Meetup 2010
Membase Meetup 2010
 
Apache Spark Based Reliable Data Ingestion in Datalake with Gagan Agrawal
Apache Spark Based Reliable Data Ingestion in Datalake with Gagan AgrawalApache Spark Based Reliable Data Ingestion in Datalake with Gagan Agrawal
Apache Spark Based Reliable Data Ingestion in Datalake with Gagan Agrawal
 
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
 
Leveraging docker for hadoop build automation and big data stack provisioning
Leveraging docker for hadoop build automation and big data stack provisioningLeveraging docker for hadoop build automation and big data stack provisioning
Leveraging docker for hadoop build automation and big data stack provisioning
 
The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)
The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)
The Future of Postgres Sharding / Bruce Momjian (PostgreSQL)
 
Reshape Data Lake (as of 2020.07)
Reshape Data Lake (as of 2020.07)Reshape Data Lake (as of 2020.07)
Reshape Data Lake (as of 2020.07)
 
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
 
Graphene – Microsoft SCOPE on Tez
Graphene – Microsoft SCOPE on Tez Graphene – Microsoft SCOPE on Tez
Graphene – Microsoft SCOPE on Tez
 
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...
 
ETL Practices for Better or Worse
ETL Practices for Better or WorseETL Practices for Better or Worse
ETL Practices for Better or Worse
 
Loading Data into Redshift: Data Analytics Week at the SF Loft
Loading Data into Redshift: Data Analytics Week at the SF LoftLoading Data into Redshift: Data Analytics Week at the SF Loft
Loading Data into Redshift: Data Analytics Week at the SF Loft
 
Real time fraud detection at 1+M scale on hadoop stack
Real time fraud detection at 1+M scale on hadoop stackReal time fraud detection at 1+M scale on hadoop stack
Real time fraud detection at 1+M scale on hadoop stack
 
From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...
From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...
From Insights to Value - Building a Modern Logical Data Lake to Drive User Ad...
 

Viewers also liked

Apache Kylin’s Performance Boost from Apache HBase
Apache Kylin’s Performance Boost from Apache HBaseApache Kylin’s Performance Boost from Apache HBase
Apache Kylin’s Performance Boost from Apache HBase
HBaseCon
 
Apache HBase at Airbnb
Apache HBase at Airbnb Apache HBase at Airbnb
Apache HBase at Airbnb
HBaseCon
 
Rolling Out Apache HBase for Mobile Offerings at Visa
Rolling Out Apache HBase for Mobile Offerings at Visa Rolling Out Apache HBase for Mobile Offerings at Visa
Rolling Out Apache HBase for Mobile Offerings at Visa
HBaseCon
 
Apache HBase in the Enterprise Data Hub at Cerner
Apache HBase in the Enterprise Data Hub at CernerApache HBase in the Enterprise Data Hub at Cerner
Apache HBase in the Enterprise Data Hub at Cerner
HBaseCon
 
Keynote: The Future of Apache HBase
Keynote: The Future of Apache HBaseKeynote: The Future of Apache HBase
Keynote: The Future of Apache HBase
HBaseCon
 
Apache HBase - Just the Basics
Apache HBase - Just the BasicsApache HBase - Just the Basics
Apache HBase - Just the Basics
HBaseCon
 
Apache Phoenix: Use Cases and New Features
Apache Phoenix: Use Cases and New FeaturesApache Phoenix: Use Cases and New Features
Apache Phoenix: Use Cases and New Features
HBaseCon
 
Galera webinar migration to galera cluster from my sql async replication
Galera webinar migration to galera cluster from my sql async replicationGalera webinar migration to galera cluster from my sql async replication
Galera webinar migration to galera cluster from my sql async replication
Codership Oy - Creators of Galera Cluster
 
Design cube in Apache Kylin
Design cube in Apache KylinDesign cube in Apache Kylin
Design cube in Apache Kylin
Yang Li
 
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCHBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
Cloudera, Inc.
 
HBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBaseHBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBase
Cloudera, Inc.
 
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
Cloudera, Inc.
 
HBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on FlashHBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on Flash
Cloudera, Inc.
 
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARNHBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon
 
Tales from the Cloudera Field
Tales from the Cloudera FieldTales from the Cloudera Field
Tales from the Cloudera Field
HBaseCon
 
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
Cloudera, Inc.
 
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
Cloudera, Inc.
 
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
Cloudera, Inc.
 
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
Cloudera, Inc.
 
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBaseHBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon
 

Viewers also liked (20)

Apache Kylin’s Performance Boost from Apache HBase
Apache Kylin’s Performance Boost from Apache HBaseApache Kylin’s Performance Boost from Apache HBase
Apache Kylin’s Performance Boost from Apache HBase
 
Apache HBase at Airbnb
Apache HBase at Airbnb Apache HBase at Airbnb
Apache HBase at Airbnb
 
Rolling Out Apache HBase for Mobile Offerings at Visa
Rolling Out Apache HBase for Mobile Offerings at Visa Rolling Out Apache HBase for Mobile Offerings at Visa
Rolling Out Apache HBase for Mobile Offerings at Visa
 
Apache HBase in the Enterprise Data Hub at Cerner
Apache HBase in the Enterprise Data Hub at CernerApache HBase in the Enterprise Data Hub at Cerner
Apache HBase in the Enterprise Data Hub at Cerner
 
Keynote: The Future of Apache HBase
Keynote: The Future of Apache HBaseKeynote: The Future of Apache HBase
Keynote: The Future of Apache HBase
 
Apache HBase - Just the Basics
Apache HBase - Just the BasicsApache HBase - Just the Basics
Apache HBase - Just the Basics
 
Apache Phoenix: Use Cases and New Features
Apache Phoenix: Use Cases and New FeaturesApache Phoenix: Use Cases and New Features
Apache Phoenix: Use Cases and New Features
 
Galera webinar migration to galera cluster from my sql async replication
Galera webinar migration to galera cluster from my sql async replicationGalera webinar migration to galera cluster from my sql async replication
Galera webinar migration to galera cluster from my sql async replication
 
Design cube in Apache Kylin
Design cube in Apache KylinDesign cube in Apache Kylin
Design cube in Apache Kylin
 
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCHBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
 
HBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBaseHBaseCon 2013: Rebuilding for Scale on Apache HBase
HBaseCon 2013: Rebuilding for Scale on Apache HBase
 
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
 
HBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on FlashHBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on Flash
 
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARNHBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
 
Tales from the Cloudera Field
Tales from the Cloudera FieldTales from the Cloudera Field
Tales from the Cloudera Field
 
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
 
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
HBaseCon 2013: Apache HBase, Meet Ops. Ops, Meet Apache HBase.
 
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
 
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
 
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBaseHBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
HBaseCon 2015: Trafodion - Integrating Operational SQL into HBase
 

Similar to In Search of Database Nirvana: Challenges of Delivering HTAP

In search of database nirvana - The challenges of delivering Hybrid Transacti...
In search of database nirvana - The challenges of delivering Hybrid Transacti...In search of database nirvana - The challenges of delivering Hybrid Transacti...
In search of database nirvana - The challenges of delivering Hybrid Transacti...
Rohit Jain
 
Embedded-ml(ai)applications - Bjoern Staender
Embedded-ml(ai)applications - Bjoern StaenderEmbedded-ml(ai)applications - Bjoern Staender
Embedded-ml(ai)applications - Bjoern Staender
Dataconomy Media
 
Empowering Real Time Patient Care Through Spark Streaming
Empowering Real Time Patient Care Through Spark StreamingEmpowering Real Time Patient Care Through Spark Streaming
Empowering Real Time Patient Care Through Spark Streaming
Databricks
 
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster AnswersR+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
Revolution Analytics
 
Options for Data Prep - A Survey of the Current Market
Options for Data Prep - A Survey of the Current MarketOptions for Data Prep - A Survey of the Current Market
Options for Data Prep - A Survey of the Current Market
Dremio Corporation
 
Comcast Labs Connect - PHLAI Conference Philadelphia 2018
Comcast Labs Connect - PHLAI Conference Philadelphia 2018 Comcast Labs Connect - PHLAI Conference Philadelphia 2018
Comcast Labs Connect - PHLAI Conference Philadelphia 2018
Open Data Group
 
Knowledge Graph for Machine Learning and Data Science
Knowledge Graph for Machine Learning and Data ScienceKnowledge Graph for Machine Learning and Data Science
Knowledge Graph for Machine Learning and Data Science
Cambridge Semantics
 
How to Automate your Enterprise Application / ERP Testing
How to Automate your  Enterprise Application / ERP TestingHow to Automate your  Enterprise Application / ERP Testing
How to Automate your Enterprise Application / ERP Testing
RTTS
 
AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...
AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...
AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...
Cambridge Semantics
 
Renewing the BI infrastructure at Hellorider - Big Data Expo 2019
Renewing the BI infrastructure at Hellorider - Big Data Expo 2019Renewing the BI infrastructure at Hellorider - Big Data Expo 2019
Renewing the BI infrastructure at Hellorider - Big Data Expo 2019
webwinkelvakdag
 
Choosing the Right Database for My Workload: Purpose-Built Databases
Choosing the Right Database for My Workload: Purpose-Built Databases Choosing the Right Database for My Workload: Purpose-Built Databases
Choosing the Right Database for My Workload: Purpose-Built Databases
AWS Germany
 
Initiate Edinburgh 2019 - Big Data Meets AI
Initiate Edinburgh 2019 - Big Data Meets AIInitiate Edinburgh 2019 - Big Data Meets AI
Initiate Edinburgh 2019 - Big Data Meets AI
Amazon Web Services
 
Driving Digital Transformation with Machine Learning in Oracle Analytics
Driving Digital Transformation with Machine Learning in Oracle AnalyticsDriving Digital Transformation with Machine Learning in Oracle Analytics
Driving Digital Transformation with Machine Learning in Oracle Analytics
Perficient, Inc.
 
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile ApproachUsing OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Kent Graziano
 
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...
Michael Rys
 
Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype
Hybrid Transactional/Analytics Processing: Beyond the Big Database HypeHybrid Transactional/Analytics Processing: Beyond the Big Database Hype
Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype
Ali Hodroj
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
Agile Testing Alliance
 
Introducing new AIOps innovations in Oracle 19c - San Jose AICUG
Introducing new AIOps innovations in Oracle 19c - San Jose AICUGIntroducing new AIOps innovations in Oracle 19c - San Jose AICUG
Introducing new AIOps innovations in Oracle 19c - San Jose AICUG
Sandesh Rao
 
Oracle Big Data Appliance and Big Data SQL for advanced analytics
Oracle Big Data Appliance and Big Data SQL for advanced analyticsOracle Big Data Appliance and Big Data SQL for advanced analytics
Oracle Big Data Appliance and Big Data SQL for advanced analytics
jdijcks
 

Similar to In Search of Database Nirvana: Challenges of Delivering HTAP (20)

In search of database nirvana - The challenges of delivering Hybrid Transacti...
In search of database nirvana - The challenges of delivering Hybrid Transacti...In search of database nirvana - The challenges of delivering Hybrid Transacti...
In search of database nirvana - The challenges of delivering Hybrid Transacti...
 
Embedded-ml(ai)applications - Bjoern Staender
Embedded-ml(ai)applications - Bjoern StaenderEmbedded-ml(ai)applications - Bjoern Staender
Embedded-ml(ai)applications - Bjoern Staender
 
Empowering Real Time Patient Care Through Spark Streaming
Empowering Real Time Patient Care Through Spark StreamingEmpowering Real Time Patient Care Through Spark Streaming
Empowering Real Time Patient Care Through Spark Streaming
 
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster AnswersR+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
R+Hadoop - Ask Bigger (and New) Questions and Get Better, Faster Answers
 
Options for Data Prep - A Survey of the Current Market
Options for Data Prep - A Survey of the Current MarketOptions for Data Prep - A Survey of the Current Market
Options for Data Prep - A Survey of the Current Market
 
Comcast Labs Connect - PHLAI Conference Philadelphia 2018
Comcast Labs Connect - PHLAI Conference Philadelphia 2018 Comcast Labs Connect - PHLAI Conference Philadelphia 2018
Comcast Labs Connect - PHLAI Conference Philadelphia 2018
 
Knowledge Graph for Machine Learning and Data Science
Knowledge Graph for Machine Learning and Data ScienceKnowledge Graph for Machine Learning and Data Science
Knowledge Graph for Machine Learning and Data Science
 
How to Automate your Enterprise Application / ERP Testing
How to Automate your  Enterprise Application / ERP TestingHow to Automate your  Enterprise Application / ERP Testing
How to Automate your Enterprise Application / ERP Testing
 
AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...
AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...
AnzoGraph DB: Driving AI and Machine Insights with Knowledge Graphs in a Conn...
 
Renewing the BI infrastructure at Hellorider - Big Data Expo 2019
Renewing the BI infrastructure at Hellorider - Big Data Expo 2019Renewing the BI infrastructure at Hellorider - Big Data Expo 2019
Renewing the BI infrastructure at Hellorider - Big Data Expo 2019
 
Choosing the Right Database for My Workload: Purpose-Built Databases
Choosing the Right Database for My Workload: Purpose-Built Databases Choosing the Right Database for My Workload: Purpose-Built Databases
Choosing the Right Database for My Workload: Purpose-Built Databases
 
Initiate Edinburgh 2019 - Big Data Meets AI
Initiate Edinburgh 2019 - Big Data Meets AIInitiate Edinburgh 2019 - Big Data Meets AI
Initiate Edinburgh 2019 - Big Data Meets AI
 
Driving Digital Transformation with Machine Learning in Oracle Analytics
Driving Digital Transformation with Machine Learning in Oracle AnalyticsDriving Digital Transformation with Machine Learning in Oracle Analytics
Driving Digital Transformation with Machine Learning in Oracle Analytics
 
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile ApproachUsing OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
Using OBIEE and Data Vault to Virtualize Your BI Environment: An Agile Approach
 
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...
 
Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype
Hybrid Transactional/Analytics Processing: Beyond the Big Database HypeHybrid Transactional/Analytics Processing: Beyond the Big Database Hype
Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
ATAGTR2017 Performance Testing and Non-Functional Testing Strategy for Big Da...
 
Introducing new AIOps innovations in Oracle 19c - San Jose AICUG
Introducing new AIOps innovations in Oracle 19c - San Jose AICUGIntroducing new AIOps innovations in Oracle 19c - San Jose AICUG
Introducing new AIOps innovations in Oracle 19c - San Jose AICUG
 
Oracle Big Data Appliance and Big Data SQL for advanced analytics
Oracle Big Data Appliance and Big Data SQL for advanced analyticsOracle Big Data Appliance and Big Data SQL for advanced analytics
Oracle Big Data Appliance and Big Data SQL for advanced analytics
 

More from HBaseCon

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
HBaseCon
 
hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beam
HBaseCon
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
HBaseCon
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
HBaseCon
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Netease
HBaseCon
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践
HBaseCon
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台
HBaseCon
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.com
HBaseCon
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
HBaseCon
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
HBaseCon
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMi
HBaseCon
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0
HBaseCon
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBase
HBaseCon
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon
 
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBaseHBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon
 
HBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBase
HBaseCon
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBase
HBaseCon
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at Didi
HBaseCon
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon
 

More from HBaseCon (20)

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
 
hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beam
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Netease
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.com
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMi
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBase
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
 
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBaseHBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
 
HBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBase
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBase
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at Didi
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase Client
 

Recently uploaded

Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
aslasdfmkhan4750
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
kayash1656
 
Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)
miso_uam
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
akshesh doshi
 
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdfIoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
mohitd6
 
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction InnovationNYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS Construction ERP Software
 
Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech
 
HIRE A HACKER FOR CHEATING HUSBAND/WIFE)
HIRE A HACKER FOR CHEATING HUSBAND/WIFE)HIRE A HACKER FOR CHEATING HUSBAND/WIFE)
HIRE A HACKER FOR CHEATING HUSBAND/WIFE)
josephinedrea942
 
How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024
TaskSprint | Employee Efficiency Software
 
Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …
908dutch
 
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
simran hot girls
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
Nextskill Technologies
 
Attendance Tracking From Paper To Digital
Attendance Tracking From Paper To DigitalAttendance Tracking From Paper To Digital
Attendance Tracking From Paper To Digital
Task Tracker
 
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
902basic
 
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
singhlata50dh
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
jealousviolet
 
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
rachitkumar09887
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
ashiklo9823
 
Introduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of ThingsIntroduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of Things
NachuSubramanian1
 

Recently uploaded (20)

Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
 
Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
 
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdfIoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
 
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction InnovationNYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction Innovation
 
Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.
 
HIRE A HACKER FOR CHEATING HUSBAND/WIFE)
HIRE A HACKER FOR CHEATING HUSBAND/WIFE)HIRE A HACKER FOR CHEATING HUSBAND/WIFE)
HIRE A HACKER FOR CHEATING HUSBAND/WIFE)
 
How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024
 
Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …
 
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
 
Attendance Tracking From Paper To Digital
Attendance Tracking From Paper To DigitalAttendance Tracking From Paper To Digital
Attendance Tracking From Paper To Digital
 
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
 
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
 
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
 
Introduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of ThingsIntroduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of Things
 

In Search of Database Nirvana: Challenges of Delivering HTAP

  • 1. In search of database nirvana The challenges of delivering Hybrid Transactional and Analytical Processing Rohit Jain, CTO rohit.jain@esgyn.com (C) Copyright 2015 Esgyn Corporation Esgyn Confidential
  • 2. Agenda The swinging database pendulum Hybrid Transaction and Analytical Processing (HTAP) Workloads Query versus storage engines The challenges of HTAP ◦ Single query engine for all workloads ◦ Supporting multiple storage engines ◦ Same data model for all workloads ◦ Enterprise-caliber capabilities Conclusion (C) Copyright 2015 Esgyn Corporation Esgyn Confidential
  • 3. The swinging database pendulum (C) Copyright 2015 Esgyn Corporation Esgyn Confidential RDBMS NoSQL • TCO • Elastic scalability • High performance • Semi-structured & unstructured data • Parallelization of user code • Schema flexibility • Modest needs Polyglot programming & persistence • graph database • document stores • text search • column stores • key value stores • wide column stores • Too many languages, interfaces, APIs, & data structures • Too much of gluing technologies together • Compatibility between different versions • No end-to-end view of workload performance • Support contracts with multiple vendors • Too many skills required to develop and manage • Too much data movement • No single solution for varied interfaces & use cases SQL • Skills prevalent • Existing tools & applications • Transaction support useful • More efficient when joins needed • Easier than coding M/R • Merit in rigor of pre-defining columns • Uniform metadata across applications
  • 4. Hybrid Transaction and Analytical Processing (HTAP) Workloads (C) Copyright 2015 Esgyn Corporation Esgyn Confidential OLTP • Mostly transactional • Sub-second response • Customer experience • Large update volume • High concurrency • Scales linearly • Normalized data model • Custom applications or 3rd party solutions • Mostly SMP; MPP for web-scale • Keyed updates/queries ODS • Can be transactional • Sub-second to seconds • Customer experience or Business internal • Batch to streaming feeds from OLTP • Low update volume • Low concurrency if internal, high otherwise • Near linear scale • Historical data • Normalized data model • Custom apps / 3rd party • Keyed queries BI • Non-transactional • Seconds to minutes • Business internal • Batch to streaming feeds from OLTP/ODS • No direct updates • Low to high concurrency • Less linear in scale • Historical data • Dimension data model • BI tools – reporting & dashboards • Ad hoc & scheduled queries and large extracts Analytics • Non-transactional • Minutes to hours • Business internal • Batch/aggregates from BI • No direct updates • Low concurrency • Complex queries, non- linear scale • Historical & big data • Columnar store • Analytics in database • Analytical tools • Ad hoc queries Essential to operate the business To improve performance of the company
  • 5. Query versus storage engines (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Hadoop Cluster Switch Switch Operational Business Intelligence Analytics Query Engine • Allow clients to connect & submit queries • Distribute connections across cluster • Compile query • Execute query • Return results of query to client Storage Engine • Storage structure • Partitioning • Automatic data repartitioning • Select columns • Select rows based on predicates • Caching writes and reads • Clustering by key • Fast access paths or filtering • Transactional support • Replication • Compression & Encryption • Mixed workload support • Bulk data ingest/extract • Indexing • Colocation or node locality • Data Governance • Security • Disaster recovery • Backup, Archive, Restore • Multi-temperate data support In-memory Single Query Engine
  • 6. The challenges of HTAP Single query engine for all workloads Data structure – key support, clustering, partitioning Statistics Predicates on non-leading or non-key columns Indexes and materialized views Degree of parallelism Reducing the search space Join type Data flow and access Mixed Workload Feature support (C) Copyright 2015 Esgyn Corporation Esgyn Confidential 80 minutes 2 minutes Equal-height histograms
  • 7. The challenges of HTAP Single query engine for all workloads Data structure – key support, clustering, partitioning Statistics Predicates on non-leading or non-key columns Indexes and materialized views Degree of parallelism Reducing the search space Join type Data flow and access Mixed Workload Feature support (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Week Item Store … 01/07/2016 1 1 … 01/07/2016 1 3 … 01/07/2016 1 5 … 01/07/2016 2 34 … 01/07/2016 3 13 … 01/07/2016 3 3 … 01/07/2016 4 2 … 01/07/2016 4 4 … 01/14/2016 1 2 … 01/14/2016 1 4 … 01/14/2016 1 5 … 01/14/2016 1 35 … 01/14/2016 3 1 … 01/14/2016 3 20 … Where is item = 1, Stores 2 through 5?
  • 8. The challenges of HTAP Single query engine for all workloads Data structure – key support, clustering, partitioning Statistics Predicates on non-leading or non-key columns Indexes and materialized views Degree of parallelism Reducing the search space Join type Data flow and access Mixed Workload Feature support (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Serial vs parallel plans Node 1 Node 2 Node n Client Application HDFS HBase Region 1 Filters HDFS HDFS HDFS HDFS Ethernet Coprocessors HBase Region 2 HBase Region 3 HBase Region 4 HBase Region 5 Master Master Multi- fragment Master ESP ESP ESP ESP ESP ESP ESP ESP ESP ESP
  • 9. The challenges of HTAP Single query engine for all workloads Data structure – key support, clustering, partitioning Statistics Predicates on non-leading or non-key columns Indexes and materialized views Degree of parallelism Reducing the search space Join type Data flow and access Mixed Workload Feature support (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Qry1 Qry2Qry4 Qry3Qry5 Qry6 Qry7
  • 10. The challenges of HTAP Single query engine for all workloads Data structure – key support, clustering, partitioning Statistics Predicates on non-leading or non-key columns Indexes and materialized views Degree of parallelism Reducing the search space Join type Data flow and access Mixed Workload Feature support (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Adaptive and parallel joins • Nested join • Probe cache for nested join • Merge join • Matching partition join • Repartitioned hash join • Replication by broadcast hash join • Inner / outer child broadcast • Dimensional schema star join • Inner join • Left Join • Right Join • Full Outer Join • Self join Cost Premiums for nested joins or serial plans
  • 11. The challenges of HTAP Single query engine for all workloads Data structure – key support, clustering, partitioning Statistics Predicates on non-leading or non-key columns Indexes and materialized views Degree of parallelism Reducing the search space Join type Data flow and access Mixed Workload Feature support (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Compute Cost Execution Environment Physical Properties Estimates Confidence Cardinality, Distribution, Correlation Sensitivity To Estimates Evaluate Risk Risk Adjustment Benefit Risk Risk Premiums • Nested join 20% • Merge join 10% • Serial plan 5%   ?
  • 12. Data structure – key support, clustering, partitioning Statistics Predicates on non-leading or non-key columns Indexes and materialized views Degree of parallelism Reducing the search space Join type Data flow and access Mixed Workload Feature support • Priority / SLA based execution • Allocation of resources by service level • Decrease priority with usage increase • Anti-starvation / switch between queries based on priority The challenges of HTAP Single query engine for all workloads (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Query Low Query Medium Queue Memstore HBase …. Memstore HBase Memstore HBase Queue Queue HBase Region 1 HBase Region 3 HBase Region 5 Query High Low Low Low Medium MediumMedium High HighHighLow Low Low Medium MediumMedium High HighHigh
  • 13. The challenges of HTAP Supporting multiple storage engines Statistics Key structure Partitioning Data type support Projection and selection Extensibility Security enforcement Transaction Management Metadata support Performance, scale, and concurrency considerations Error handling Other operational aspects (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Single-Master Multiple-Masters
  • 14. The challenges of HTAP Same data model for all workloads (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Normal form • 1NF • 2NF • 3NF • BCNF • 4NF • 5NF • 6NF Star Schema Snowflake Schema Normal Form Query engine integration with storage engine(s) to support all these data models
  • 15. The challenges of HTAP Same data model for all workloads (C) Copyright 2015 Esgyn Corporation Esgyn Confidential NoSQL Data Models “NoSQL Data Modeling Techniques” by Ilya Katsov Highly Scalable Blog … and these!
  • 16. The challenges of HTAP Enterprise-caliber capabilities High Availability Security Manageability (C) Copyright 2015 Esgyn Corporation Esgyn Confidential • Percentage of uptime 99.99% = 52.56 minutes downtime to 99.999% = 5.26 • Online operations (data available for reads and writes) o Upgrading the OS o Upgrading the file system o Upgrading the storage engine o Upgrading the query engine o Redistribute data to accommodate node and/or disk expansions and contractions o Changing table definition, e.g. data type changes, and adding, dropping, renaming columns o Create/drop secondary indexes o Full and Incremental Backups
  • 17. The challenges of HTAP Enterprise-caliber capabilities High Availability Security Manageability (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Schema Management Performance Management Monitoring Security Management BAR Management Object Management Performance Monitoring Database Monitor User Management Backup Analysis Graphical Object Editor Live Performance Monitoring Event Monitoring Role Management Recovery Cross-Platform Schema Knowledge Data Repository Live Event Monitoring Account Migration Log Backup Bottleneck Analysis Threshold Alerts Audit Report Backup Reports SQL Management Job/Workload Analysis Health Index Alarm Archival Query Builder Job/Workload Wizard Live Health Monitoring Visual Difference Tool Job/Workload Management Response Times Maintenance Configuration Management Data Management Live Job/Workload Monitoring Alert Center Repository Aging OS Provisioning Data Migration OS Analysis Remote Monitoring Automated Maintenance Cluster Provisioning SQL Profiler Capacity Capture Central Monitoring Instance Provisioning Automated Import Capacity Trending Hardware Inventory Change Management Cloud Provisioning Visual Explain Plans Capacity Forecast Hardware Monitoring Schema Capture Configuration Editor Session Management Space Management Schema Compare and Synch Lock Management Reorganization Management Troubleshooting Notifications Process Management Query Cost Simulation Health Analysis Schema Rotation Consistency Checks Historical Reports Problem Correlation Collaboration Online Schema Evolution Bottleneck Tuning Automated Actions Virtual Changes Built-In Automation Access Path Analysis
  • 18. The challenges of HTAP Enterprise-caliber capabilities High Availability Security Manageability (C) Copyright 2015 Esgyn Corporation Esgyn Confidential • Operational performance by transactions per second • Analytical performance by query • Overhead of gathering metrics on operational and analytical workloads • Configurable statistics collection • Workload management by Service Level Objectives o Based on priority and/or resource allocation o High priority operational workloads vs analytical workloads • End-to-end visibility of transaction and query metrics • Metric breakdown down to the query operation • Metrics for table access across workloads down to the partition level • Skew or bottlenecks • Integration with YARN
  • 19. Conclusion (C) Copyright 2015 Esgyn Corporation Esgyn Confidential Pre-register for full O’Reilly report: http://www.oreilly.com/go/dbnirvana It ain’t easy!! Very few products can even come close Any guesses? 