SlideShare a Scribd company logo
1 of 53
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Trafodion
Enterprise class Transactional SQLon Hadoop
by
Krishna Kumar, Architect
Karthikeyan Soundararajan, Architect
Open Source India 2014 – November 8th
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
Agenda
+
Motivation – Why?
Overview – What?
Use Cases – Where?
Architecture
Demo
Backup Slides
Open Source – How?
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
+MOTIVATION:
Why Trafodion?
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
+
… and limitations
Query
Optimization
Data
Integrity
Workload
Management
Transaction
Support
Real-time
Performance
Hadoop has strengths…
Social media Video Audio
Email
ImagesTexts
Documents Mobile
Offline
Analytics
Data “Dumping”
Scalability
Replication/K-safety
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
+
Introducing Trafodion:
Transactional SQL on Hadoop
Full Hadoop
support
Scales like Hadoop Extends
HAVEn
HAVEn
Social media Video Audio
Email Texts Mobile
ImagesDocuments
Transactional
data
Adds enterprise-class
transactional and reporting
functionality with full SQL
Workload
Management
Transaction
Support
Transaction
Support
Real-time
Performance
Data
Integrity
Query
Optimization
Multi-
structured
Data
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
+
Overview:
What is
Trafodion?
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Trafodion - Introduction
Complete: Full-function ANSI SQL
Reuse existing SQL skills and improve developer productivity
Protected: Distributed ACID transactions
Guarantees data consistency across multiple rows, tables, SQL statements
Efficient: Optimized for low-latency read and write
transactions
Supports real-time transaction processing applications
Flexible: Schema flexibility and multi-structured data
Seamlessly integrates structured, unstructured, and semi-structured data
Interoperable: Standard ODBC/JDBC access
Works with existing tools and applications
Open: Hadoop and Linux distribution neutral
Easy to add to your existing infrastructure and no vendor lock-in
+
Transactional
SQL
Hadoop
Open source project to develop transactional SQL on HBase
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Trafodion innovation built upon Hadoop stack
Leverages Hadoop and
HBase for core modules
Maintains API
compatibility
Differentiation
• ANSI SQL via ODBC/JDBC
• Relational schema abstraction
• Distributed transaction protection
• Low latency reads and writes
• Automatic parallelism
Hadoop Trafodion
Client Application using
ODBC/JDBC on
Windows/Linux
HBase
Hive
HDFS
Zookeeper
SQL Compiler / Optimizer / Executor
Distributed Transaction Manager
Client Services for ODBC and JDBC
+
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
+
Use Cases:
Where
Trafodion is
used?
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Potential Use case Profile
• Online
financial
management
Finance
• Billing
systems
• Provisioning
systems
Telecom
• RFID
tracking
Manufacturing
• Smart
Metering
Energy
• Authorization
and claims
processing
Healthcare
• 911
Emergency
System
Government
• Reservation
systems
Transportation
• Online
shopping
Consumer &
Retail
Multi-structured
data requirements
HBase – but through
SQL with standard
tools
Generates Revenue Touches the Customer Helps Run the Business
Experiencing scalability
or prohibitive licensing
issues
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Some projects ongoing…
+
Aimed to continuously measure the performance of all SEM Campaigns. The Engine extracts information using Google
Ad words API and provides a real time view of some of the key parameters like Quality Score (QS), Click through Rate
(CTR), Impression Share (IS), Cost Per Click (CPC), Cost Per Acquisition (CPA), Cost Per Download (CPD), Average
Position, and more, thereby enabling the Business & Marketing Functions to take real-time decisions. In the current
scenario the TAT (Turnaround Time) is anywhere between few weeks to a month, which affects the campaign
performance, leading to low conversions, poor lead quality & high marketing investments.
Enterprise Software company in China specializing in commercial fleet telemetry. Use telemetry from buses to make
them safer and more efficient to operate. Currently monitor 65K buses. Steady growth, was 10K buses on Jan 2013
Telemetry data to be stored in Trafodion tables for short latency access. Periodic ETL to Vertica for analysis
Data ingest at about 10K – 30K rows per second. Concurrent access of data though relatively simple SELECT queries, at
high concurrency, with sub-second response times
OSS allows business users and partners to track status of orders and run scheduled and ad-hoc reports. This application
has a need to move to Hadoop to save on S/W license cost.
HP IceWall SSO is designed to adapt to various customers’ environments. It has a lot of parameters, templates and APIs
in order to fulfill many kinds of customers’ requirements. And it is designed so that these customizations will not affect
any future upgrades. HP IceWall SSO can flexibly connect using many types of authorization used in web applications. In
particular it deals with 11 methods and 48 patterns of Form AuthN.
Its latest version, HP IceWall SSO 10.0, now provides support for new leading-edge technologies such as cloud and
virtualization, and the IceWall SSO product line has been extended to include Windows support in addition to the existing
HP-UX and Linux versions.
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
+
Open
Source:
How
Trafodion is
used?
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Required Software
Software Version HBase Version
Linux 6.3,6.4 kernel
Zookeeper V3.4.5
Trafodion 0.9.0
One of the following Hadoop Distribution:
- Cloudera CDH 4.5 0.94.6
CDH 5.1.2 0.98
- Hortonworks HDP 1.3.3 0.94.6
HDP 2.1 0.98
- MapR M5 v3 0.94.13/0.94.17
Platforms
Cloud
VM
Workstation
Cluster
Trafodion Installation Link -
https://wiki.trafodion.org/wiki/index.php/Installation
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Modern open source environment
Source code in GitHub
Build/test in OpenStack gerrit, zuul, jenkins
Defect tracking in LaunchPad
Documentation in MediaWiki
Following best practices of OpenStack project
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Building an Open Community
Simple installation
Meritocracy
Share your expertise: Developing, fixing defects,
testing, writing, translating and more
Seeking early adopters
Recruiting project contributors
Discover our capabilities: Download and install in
your Hadoop environment and take a test-drive
www.trafodion.org
Email:
Generic Query- Project.Trafodion@hp.com
Future directions – hema.ramaswamy@hp.com
(Hema Ramaswamy, HP Labs)
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
+
High Level
Architecture
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
3-Layered High Level Architecture
Client
JDBC ODBC
User and ISV Operational
Applications
Driver
Hive
Native Hive Tables
Multi-Structured
Data Store
Integration
HBase
Native HBase
Tables KVS,
Columnar
SQL
ESP
CMP Master
ESPDTM
WMS
Compiler and Optimizer
Workload Management(wip)
SQL Parallelism
Distributed Transaction
Management
. . . .
Database Connectivity
UDF
Communicate with
external processes
HBase
HDFS
Relational
Schema
Trafodion
Tables
Storage
Engines
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
Connectivity Architecture (T4)
Client
User and ISV
Operational
Applications
ODBC/JDBC
Drivers
SQL
DCS
Master ZooKeeper
Master
Executor
Master
Executor
Master
Executor
Master
ExecutorMaster
ExecutorMaster
Executor
DCS
Server
Database
Connectivity
Services
. . .
.
Blue: control flow
Green: data flow
Connection
Mgmt
Process
Mgmt
DCS
Server
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19
Process Architecture
Trafodion Node
(DCS,EXE, ESP, CMP, DTM,
UDF, WMS)
VM or Physical Node
Hadoop
Data Node
HBase APIs
VM or Physical Node
TCP/IP
TCP/IP
…
Trafodion
Metadata
Trafodion
Data
HBase Region
Server
Hive/HDFS
APIs
Hive
Data
HDFS
Data
Trafodion Node
(DCS,EXE, ESP, CMP, DTM,
UDF, WMS)
Hadoop
Data Node
HBase Region
Server
Hive/HDFS
APIs
HDFS
Data
Hive
Data
Trafodion
Data
Trafodion
Metadata
HBase APIs
TCP/IP
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20
Operational workload optimizations
• Key-based access with SQL “pushdown”
• Statistics based plan generation
• Query plan caching
• Data-flow, scheduler driven executor
• DOP optimization
• Parallelism without map-reduce
• Secondary index support
• Table structure optimizations
Data-flow, Scheduler-driven Salting of Row Keys
Optimized DOPStatistics Based Optimizations
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
+
Demo
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22
Enterprise-class SQL DBMS that uses native Hadoop formats (e.g., HBase, Hive,
HDFS) for data storage
Full-functioned ANSI SQL language support
Standard ODBC/JDBC connectivity for Linux and Windows clients
Low latency reads and writes via compile time and run time optimizations
ACID distributed transaction protection over multiple stmts, tables, rows
Support for big data sets using parallel SQL optimizations
Retention of Hadoop benefits: reduced cost, scalability, elasticity and data
redundancy
Support for structured, unstructured, semi-structured data and flexible schemas
Major Features - Recap
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
See for yourself…
Come discover and develop on Trafodion
www.trafodion.org
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank You
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Backup Slides
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26
External
Features
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27
Trafodion DDL
CREATE/DROP/ALTER statements
• Tables, views, indexes, columns
• Numeric, character, varchar, date, time, interval
• Unicode (UTF8, UCS2) and single byte (ISO8859-1) for user data
• UTF8 for metadata
• Salting (Table partitioning for uniform data access across disks)
LOBs
• BLOB/CLOB datatypes (wip)
Constraints
• RI, Foreign Key, Primary Key, Check
Security, Privileges
• Grant/Revoke, create/drop user
SPJ/UDF
• Java Stored Procedures
• User Defined Functions (wip)
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.28
Trafodion DML
Query statements
• SELECT, INSERT, UPDATE, DELETE, UPSERT and MERGE
Complex SQL operations
• JOIN (INNER, LEFT/RIGHT/FULL OUTER), UNION, WHERE, GROUP
BY, HAVING, ORDER BY, SAMPLING, TRANSPOSE, GROUP PIVOT,
etc.
Compile time and run time optimizations
• Cost-based query optimizer, MDAM, OR optimizations
• Correlated and nested subqueries
Cursor support (non-holdable)
SQL functions
• Aggregate, date/time, character, mathematical, OLAP, sequence, etc.
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.29
Miscellaneous
Utilities
• Update Statistics
• Explain
• Control Query Shape, Control Query Default
Transaction Control
• BEGIN WORK, COMMIT WORK, ROLLBACK WORK
• SET TRANSACTION
Oracle compatibility
• Based on internal POCs
• Syntax extensions (DUAL, ROWNUM, SYSDATE, NEXTVAL…)
• Functions (TO_CHAR/DATE, SEQUENCE, Incompatible
operations, …)
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.30
Trafodion Interfaces
Product interfaces
• TrafCI
• sqlci
• Trafodion Query Workbench
• ODBC client app
• JDBC client app
• T4 Driver
• T2 Driver (wip)
• HP DSM (wip)
External Interfaces
• DBVisualizer
• SQuirrel
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.31
Multiple Storage Engines
Trafodion
• Uses hbase tables underneath
• native datatype storage format (ex: 2 bytes for a short)
• Encoded data for serialization
• column family and column name optimization
Native Hbase
• Cell format: One hbase cell per row output
• Rowwise format: All cells in one row
Hive
• External metadata from hive
• Text files: delimited data
• Sequence files: structured data
• ORC files: Optimized Row Columnar data (wip)
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.32
Interoperability
Trafodion, Hbase, Hive
• Syntax extension to identify tables
• Trafodion.sch.t_traf
• Hbase.”_CELL_”.t_hbase
• hive.hive.t_hive
JOINs, INSERT…SELECT
Transactional updates across storages
• Currently Trafodion and HBase
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.33
External integration
• Table-valued UDF
• Data Loader
• Data Extractor
• Other products through Table-valued UDF (wip)
WMS (Workload Management)
• Runtime stats data collection
• Repository updates (wip)
• Automated query control (wip)
• Resource control (wip)
• User control (wip)
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.34
DISTRIBUT
ED
TRANSACTI
ON
MANAGEME
NT
(DTM)
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.35
Overview
Single row consistency using underlying HBase
Global transactions across multiple tables/rows
Optimistic locking
• Conflict resolution at commit
• First commit gets through
HBase Trx EndPoint Coprocessors
HBase Write Ahead Log (HLOG) for audit logging
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.36
...
Node 1
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
Transaction
Manager
HBase trx Region Server
Node 2
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
Transaction
Manager
HBase trx Region Server
Node n
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
SQL Process
Transaction
Manager
Library
Resource
Manager
Library
Transaction
Manager
HBase trx EndPoint
Coprocessor
Distributed, Scalable Architecture
HBase trx EndPoint
Coprocessor
HBase trx EndPoint
Coprocessor
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.38
Performance
Benchmarks
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.39
Configuration for Perf Runs
System under Test: Spinel ( Converted Gen8 Seaquest Cluster)
Spinel is not an optimal configuration for Trafodion/Hbase due to the single raided data
drive.
Nodes: 10; Memory: 128 Gibper node; Cores: 16 per node; OS:
RedhatLinux 6.3; Data Drives: 1 per node ( 3.4 TB Raid device)
Hadoop/Hbase: Cloudera 4.5.0; Hadoop 2.0.0; Hbase 0.94.6/0.98
Trafodion Version 0.8.3
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.40
YCSB (Yahoo Cloud Serving Benchmark)
Workload A: read/update ratio 50/50
Traf 0.9.0
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.41
DebitCredit
5 SQL statements per transaction
(3 update, 1 insert, 1 select)
Traf
0.9.0
Traf
0.9.0
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.42
WORKLOA
D
OPTIMIZA
TIONS
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.43
Key-based access with SQL “pushdown”
Statistics based plan generation
Parallelism without map-reduce
DOP(Degree Of Parallelism) optimization
Adaptive segmentation
Query plan caching
Data-flow, scheduler driven executor
Secondary index support
Table structure optimizations
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.44
Optimized DOPStatistics Based
Optimizations
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.45
Data-flow, Scheduler-
driven
Salting of Row Keys
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.46
Real-Time Mixed Workload
Concurrent transactional OpenCart application
Queries against the same database
Query type: OLTP Insert, Deletes, Selects
Query monitoring
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.47
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.48
Operational Reporting Queries
Complex multi-table operations
Parallel query execution
Various query optimizations for
complex queries
EXPLAIN: display query plan
EXECUTE: run query
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.49
Query: List of all products, some
product info, current specials, a
summary of their ratings and
reviewsNested Join for
keyed lookup
into Trafodion
Parallel scan larger
Trafodion tables
Cache of
previous
lookups into
Trafodion
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.50
Interoperability between Storages
Access multiple Hadoop Storages
Load/Extract data from Trafodion to/from Hive/HDFS
Parallel insert and select operations
Join tables from multiple storages
Without external federation
Storage access from within engine
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.51
Load data from Trafodion tables
to Hive table with insert-select
statement
Source data is detailed order
information obtained by joining
multiple Trafodion tables
Parallel
Join
Trafodion
tables acting
as source
Parallel
insert into
Hive
Hive table
is the
target
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.52
Transactional Capabilities
Multiple rows
Multiple tables
Coordinated Commit and Rollback
Conflict resolution
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.53
TraFoto: Structured/Unstructured Integration
Access structured relational Trafodion table
Access unstructured native HBase table
JOINs and other operations between the tables
Transactional insert, update, deletes across both
tables
HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.54
UDFs: User Defined Functions
User written scalar and tabular
functions
SQL syntax to access UDFs in queries
Parallel execution at runtime

More Related Content

What's hot

Big Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo Clinic
Big Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo ClinicBig Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo Clinic
Big Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo Clinic
DataWorks Summit
 
Harnessing Big Data in Real-Time
Harnessing Big Data in Real-TimeHarnessing Big Data in Real-Time
Harnessing Big Data in Real-Time
DataWorks Summit
 
Evolving Hadoop into an Operational Platform with Data Applications
Evolving Hadoop into an Operational Platform with Data ApplicationsEvolving Hadoop into an Operational Platform with Data Applications
Evolving Hadoop into an Operational Platform with Data Applications
DataWorks Summit
 

What's hot (20)

Building Information Platform - Integration of Hadoop with SAP HANA and HANA ...
Building Information Platform - Integration of Hadoop with SAP HANA and HANA ...Building Information Platform - Integration of Hadoop with SAP HANA and HANA ...
Building Information Platform - Integration of Hadoop with SAP HANA and HANA ...
 
BDM39: HP Vertica BI: Sub-second big data analytics your users and developers...
BDM39: HP Vertica BI: Sub-second big data analytics your users and developers...BDM39: HP Vertica BI: Sub-second big data analytics your users and developers...
BDM39: HP Vertica BI: Sub-second big data analytics your users and developers...
 
The Future of Apache Hadoop an Enterprise Architecture View
The Future of Apache Hadoop an Enterprise Architecture ViewThe Future of Apache Hadoop an Enterprise Architecture View
The Future of Apache Hadoop an Enterprise Architecture View
 
The DAP - Where YARN, HBase, Kafka and Spark go to Production
The DAP - Where YARN, HBase, Kafka and Spark go to ProductionThe DAP - Where YARN, HBase, Kafka and Spark go to Production
The DAP - Where YARN, HBase, Kafka and Spark go to Production
 
Big Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo Clinic
Big Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo ClinicBig Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo Clinic
Big Data Platform Processes Daily Healthcare Data for Clinic Use at Mayo Clinic
 
Pivotal HAWQ 소개
Pivotal HAWQ 소개Pivotal HAWQ 소개
Pivotal HAWQ 소개
 
How can Hadoop & SAP be integrated
How can Hadoop & SAP be integratedHow can Hadoop & SAP be integrated
How can Hadoop & SAP be integrated
 
Ingesting Data at Blazing Speed Using Apache Orc
Ingesting Data at Blazing Speed Using Apache OrcIngesting Data at Blazing Speed Using Apache Orc
Ingesting Data at Blazing Speed Using Apache Orc
 
How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
How Big Data and Hadoop Integrated into BMC ControlM at CARFAXHow Big Data and Hadoop Integrated into BMC ControlM at CARFAX
How Big Data and Hadoop Integrated into BMC ControlM at CARFAX
 
Harnessing Big Data in Real-Time
Harnessing Big Data in Real-TimeHarnessing Big Data in Real-Time
Harnessing Big Data in Real-Time
 
Big Data Simplified - Is all about Ab'strakSHeN
Big Data Simplified - Is all about Ab'strakSHeNBig Data Simplified - Is all about Ab'strakSHeN
Big Data Simplified - Is all about Ab'strakSHeN
 
HAWQ: a massively parallel processing SQL engine in hadoop
HAWQ: a massively parallel processing SQL engine in hadoopHAWQ: a massively parallel processing SQL engine in hadoop
HAWQ: a massively parallel processing SQL engine in hadoop
 
SAP HORTONWORKS
SAP HORTONWORKSSAP HORTONWORKS
SAP HORTONWORKS
 
Hadoop crash course workshop at Hadoop Summit
Hadoop crash course workshop at Hadoop SummitHadoop crash course workshop at Hadoop Summit
Hadoop crash course workshop at Hadoop Summit
 
Internet of things Crash Course Workshop
Internet of things Crash Course WorkshopInternet of things Crash Course Workshop
Internet of things Crash Course Workshop
 
Integration of SAP HANA with Hadoop
Integration of SAP HANA with HadoopIntegration of SAP HANA with Hadoop
Integration of SAP HANA with Hadoop
 
Hortonworks Data In Motion Series Part 4
Hortonworks Data In Motion Series Part 4Hortonworks Data In Motion Series Part 4
Hortonworks Data In Motion Series Part 4
 
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...
 
Evolving Hadoop into an Operational Platform with Data Applications
Evolving Hadoop into an Operational Platform with Data ApplicationsEvolving Hadoop into an Operational Platform with Data Applications
Evolving Hadoop into an Operational Platform with Data Applications
 
Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...
Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...
Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...
 

Similar to Trafodion – an enterprise class sql based on hadoop

Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUGReal-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
skumpf
 
Carpe Datum: Building Big Data Analytical Applications with HP Haven
Carpe Datum: Building Big Data Analytical Applications with HP HavenCarpe Datum: Building Big Data Analytical Applications with HP Haven
Carpe Datum: Building Big Data Analytical Applications with HP Haven
DataWorks Summit
 
Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...
Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...
Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...
Innovative Management Services
 

Similar to Trafodion – an enterprise class sql based on hadoop (20)

Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUGReal-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
 
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
 
Storm Demo Talk - Colorado Springs May 2015
Storm Demo Talk - Colorado Springs May 2015Storm Demo Talk - Colorado Springs May 2015
Storm Demo Talk - Colorado Springs May 2015
 
How to use Hadoop for operational and transactional purposes by RODRIGO MERI...
 How to use Hadoop for operational and transactional purposes by RODRIGO MERI... How to use Hadoop for operational and transactional purposes by RODRIGO MERI...
How to use Hadoop for operational and transactional purposes by RODRIGO MERI...
 
A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...
 
HP Helion OpenStack and Professional Services
HP Helion OpenStack and Professional ServicesHP Helion OpenStack and Professional Services
HP Helion OpenStack and Professional Services
 
Carpe Datum: Building Big Data Analytical Applications with HP Haven
Carpe Datum: Building Big Data Analytical Applications with HP HavenCarpe Datum: Building Big Data Analytical Applications with HP Haven
Carpe Datum: Building Big Data Analytical Applications with HP Haven
 
Level Up – How to Achieve Hadoop Acceleration
Level Up – How to Achieve Hadoop AccelerationLevel Up – How to Achieve Hadoop Acceleration
Level Up – How to Achieve Hadoop Acceleration
 
HP Helion Webinar #1 - Introduction to HP Helion OpenStack w/Christian Frank
HP Helion Webinar #1 - Introduction to HP Helion OpenStack w/Christian FrankHP Helion Webinar #1 - Introduction to HP Helion OpenStack w/Christian Frank
HP Helion Webinar #1 - Introduction to HP Helion OpenStack w/Christian Frank
 
Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...
Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...
Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...
 
Storm Demo Talk - Denver Apr 2015
Storm Demo Talk - Denver Apr 2015Storm Demo Talk - Denver Apr 2015
Storm Demo Talk - Denver Apr 2015
 
Hadoop as an Analytic Platform: Why Not?
Hadoop as an Analytic Platform: Why Not?Hadoop as an Analytic Platform: Why Not?
Hadoop as an Analytic Platform: Why Not?
 
Hortonworks and Red Hat Webinar_Sept.3rd_Part 1
Hortonworks and Red Hat Webinar_Sept.3rd_Part 1Hortonworks and Red Hat Webinar_Sept.3rd_Part 1
Hortonworks and Red Hat Webinar_Sept.3rd_Part 1
 
Introduction to Hadoop
Introduction to HadoopIntroduction to Hadoop
Introduction to Hadoop
 
Hortonworks and Red Hat Webinar - Part 2
Hortonworks and Red Hat Webinar - Part 2Hortonworks and Red Hat Webinar - Part 2
Hortonworks and Red Hat Webinar - Part 2
 
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-HadoopHP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
 
4. Big data & analytics HP
4. Big data & analytics HP4. Big data & analytics HP
4. Big data & analytics HP
 
iKariera 2015
iKariera 2015iKariera 2015
iKariera 2015
 
Big Data & SQL: The On-Ramp to Hadoop
Big Data & SQL: The On-Ramp to Hadoop Big Data & SQL: The On-Ramp to Hadoop
Big Data & SQL: The On-Ramp to Hadoop
 
Building a Modern Data Architecture with Enterprise Hadoop
Building a Modern Data Architecture with Enterprise HadoopBuilding a Modern Data Architecture with Enterprise Hadoop
Building a Modern Data Architecture with Enterprise Hadoop
 

More from Krishna-Kumar

More from Krishna-Kumar (20)

SODA Ambassadors & Community Ecosystem
SODA Ambassadors & Community EcosystemSODA Ambassadors & Community Ecosystem
SODA Ambassadors & Community Ecosystem
 
Open Source Building Career and Competency
Open Source Building Career and CompetencyOpen Source Building Career and Competency
Open Source Building Career and Competency
 
CCICI CIP 1.0 Testbed - Security access implementation and reference - v1.0
CCICI CIP 1.0 Testbed - Security access implementation and reference - v1.0CCICI CIP 1.0 Testbed - Security access implementation and reference - v1.0
CCICI CIP 1.0 Testbed - Security access implementation and reference - v1.0
 
Google Anthos - Azure Stack - AWS Outposts :Comparison
Google Anthos - Azure Stack - AWS Outposts :ComparisonGoogle Anthos - Azure Stack - AWS Outposts :Comparison
Google Anthos - Azure Stack - AWS Outposts :Comparison
 
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAPCloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
 
Cloud interoperability and open standards for digital india open infrasummit
Cloud interoperability and open standards for digital india open infrasummitCloud interoperability and open standards for digital india open infrasummit
Cloud interoperability and open standards for digital india open infrasummit
 
Google Cloud Container Security Quick Overview
Google Cloud Container Security Quick OverviewGoogle Cloud Container Security Quick Overview
Google Cloud Container Security Quick Overview
 
Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!
 
KubeCon + CloudNativeCon Barcelona and Shanghai 2019 - Highlights
KubeCon + CloudNativeCon Barcelona and Shanghai 2019 - HighlightsKubeCon + CloudNativeCon Barcelona and Shanghai 2019 - Highlights
KubeCon + CloudNativeCon Barcelona and Shanghai 2019 - Highlights
 
Introduction to ieee standards development - Bangalore Section
Introduction to ieee standards development - Bangalore SectionIntroduction to ieee standards development - Bangalore Section
Introduction to ieee standards development - Bangalore Section
 
IEEE Standards Association - Introduction
IEEE Standards Association - IntroductionIEEE Standards Association - Introduction
IEEE Standards Association - Introduction
 
IoTShow.in Bangalore 2019 - a Recap on 'IoT and Edge' Talk.
IoTShow.in Bangalore 2019 - a Recap on 'IoT and Edge' Talk.IoTShow.in Bangalore 2019 - a Recap on 'IoT and Edge' Talk.
IoTShow.in Bangalore 2019 - a Recap on 'IoT and Edge' Talk.
 
Kubecon seattle 2018 recap - Application Deployment aspects
Kubecon seattle 2018 recap - Application Deployment aspectsKubecon seattle 2018 recap - Application Deployment aspects
Kubecon seattle 2018 recap - Application Deployment aspects
 
Open Source Edge Computing Platforms - Overview
Open Source Edge Computing Platforms - OverviewOpen Source Edge Computing Platforms - Overview
Open Source Edge Computing Platforms - Overview
 
cncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetescncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetes
 
Evolution of containers to kubernetes
Evolution of containers to kubernetesEvolution of containers to kubernetes
Evolution of containers to kubernetes
 
My Ladakh Marathon Run 2018
My Ladakh Marathon Run 2018My Ladakh Marathon Run 2018
My Ladakh Marathon Run 2018
 
Containers and workload security an overview
Containers and workload security an overview Containers and workload security an overview
Containers and workload security an overview
 
Now yoga - a study on where why what how
Now yoga  - a study on where why what howNow yoga  - a study on where why what how
Now yoga - a study on where why what how
 
CNCF Introduction - Feb 2018
CNCF Introduction - Feb 2018CNCF Introduction - Feb 2018
CNCF Introduction - Feb 2018
 

Recently uploaded

Simplify hybrid data integration at an enterprise scale. Integrate all your d...
Simplify hybrid data integration at an enterprise scale. Integrate all your d...Simplify hybrid data integration at an enterprise scale. Integrate all your d...
Simplify hybrid data integration at an enterprise scale. Integrate all your d...
varanasisatyanvesh
 
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
acoha1
 
Displacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second DerivativesDisplacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second Derivatives
23050636
 
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
siskavia95
 
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
acoha1
 
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
mikehavy0
 
如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证
如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证
如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证
zifhagzkk
 
一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样
一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样
一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样
yhavx
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
Amil baba
 
Audience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptxAudience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptx
Stephen266013
 

Recently uploaded (20)

SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarjSCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
 
Simplify hybrid data integration at an enterprise scale. Integrate all your d...
Simplify hybrid data integration at an enterprise scale. Integrate all your d...Simplify hybrid data integration at an enterprise scale. Integrate all your d...
Simplify hybrid data integration at an enterprise scale. Integrate all your d...
 
Las implicancias del memorándum de entendimiento entre Codelco y SQM según la...
Las implicancias del memorándum de entendimiento entre Codelco y SQM según la...Las implicancias del memorándum de entendimiento entre Codelco y SQM según la...
Las implicancias del memorándum de entendimiento entre Codelco y SQM según la...
 
Digital Transformation Playbook by Graham Ware
Digital Transformation Playbook by Graham WareDigital Transformation Playbook by Graham Ware
Digital Transformation Playbook by Graham Ware
 
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
 
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
 
Displacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second DerivativesDisplacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second Derivatives
 
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
 
The Significance of Transliteration Enhancing
The Significance of Transliteration EnhancingThe Significance of Transliteration Enhancing
The Significance of Transliteration Enhancing
 
Chapter 1 - Introduction to Data Mining Concepts and Techniques.pptx
Chapter 1 - Introduction to Data Mining Concepts and Techniques.pptxChapter 1 - Introduction to Data Mining Concepts and Techniques.pptx
Chapter 1 - Introduction to Data Mining Concepts and Techniques.pptx
 
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
 
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
 
如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证
如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证
如何办理(Dalhousie毕业证书)达尔豪斯大学毕业证成绩单留信学历认证
 
一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样
一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样
一比一原版(Monash毕业证书)莫纳什大学毕业证原件一模一样
 
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
 
Fuel Efficiency Forecast: Predictive Analytics for a Greener Automotive Future
Fuel Efficiency Forecast: Predictive Analytics for a Greener Automotive FutureFuel Efficiency Forecast: Predictive Analytics for a Greener Automotive Future
Fuel Efficiency Forecast: Predictive Analytics for a Greener Automotive Future
 
Genuine love spell caster )! ,+27834335081) Ex lover back permanently in At...
Genuine love spell caster )! ,+27834335081)   Ex lover back permanently in At...Genuine love spell caster )! ,+27834335081)   Ex lover back permanently in At...
Genuine love spell caster )! ,+27834335081) Ex lover back permanently in At...
 
Audience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptxAudience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptx
 
社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token Prediction社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token Prediction
 

Trafodion – an enterprise class sql based on hadoop

  • 1. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Trafodion Enterprise class Transactional SQLon Hadoop by Krishna Kumar, Architect Karthikeyan Soundararajan, Architect Open Source India 2014 – November 8th
  • 2. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2 Agenda + Motivation – Why? Overview – What? Use Cases – Where? Architecture Demo Backup Slides Open Source – How?
  • 3. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3 +MOTIVATION: Why Trafodion?
  • 4. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4 + … and limitations Query Optimization Data Integrity Workload Management Transaction Support Real-time Performance Hadoop has strengths… Social media Video Audio Email ImagesTexts Documents Mobile Offline Analytics Data “Dumping” Scalability Replication/K-safety
  • 5. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5 + Introducing Trafodion: Transactional SQL on Hadoop Full Hadoop support Scales like Hadoop Extends HAVEn HAVEn Social media Video Audio Email Texts Mobile ImagesDocuments Transactional data Adds enterprise-class transactional and reporting functionality with full SQL Workload Management Transaction Support Transaction Support Real-time Performance Data Integrity Query Optimization Multi- structured Data
  • 6. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6 + Overview: What is Trafodion?
  • 7. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 Trafodion - Introduction Complete: Full-function ANSI SQL Reuse existing SQL skills and improve developer productivity Protected: Distributed ACID transactions Guarantees data consistency across multiple rows, tables, SQL statements Efficient: Optimized for low-latency read and write transactions Supports real-time transaction processing applications Flexible: Schema flexibility and multi-structured data Seamlessly integrates structured, unstructured, and semi-structured data Interoperable: Standard ODBC/JDBC access Works with existing tools and applications Open: Hadoop and Linux distribution neutral Easy to add to your existing infrastructure and no vendor lock-in + Transactional SQL Hadoop Open source project to develop transactional SQL on HBase
  • 8. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8 Trafodion innovation built upon Hadoop stack Leverages Hadoop and HBase for core modules Maintains API compatibility Differentiation • ANSI SQL via ODBC/JDBC • Relational schema abstraction • Distributed transaction protection • Low latency reads and writes • Automatic parallelism Hadoop Trafodion Client Application using ODBC/JDBC on Windows/Linux HBase Hive HDFS Zookeeper SQL Compiler / Optimizer / Executor Distributed Transaction Manager Client Services for ODBC and JDBC +
  • 9. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9 + Use Cases: Where Trafodion is used?
  • 10. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10 Potential Use case Profile • Online financial management Finance • Billing systems • Provisioning systems Telecom • RFID tracking Manufacturing • Smart Metering Energy • Authorization and claims processing Healthcare • 911 Emergency System Government • Reservation systems Transportation • Online shopping Consumer & Retail Multi-structured data requirements HBase – but through SQL with standard tools Generates Revenue Touches the Customer Helps Run the Business Experiencing scalability or prohibitive licensing issues
  • 11. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11 Some projects ongoing… + Aimed to continuously measure the performance of all SEM Campaigns. The Engine extracts information using Google Ad words API and provides a real time view of some of the key parameters like Quality Score (QS), Click through Rate (CTR), Impression Share (IS), Cost Per Click (CPC), Cost Per Acquisition (CPA), Cost Per Download (CPD), Average Position, and more, thereby enabling the Business & Marketing Functions to take real-time decisions. In the current scenario the TAT (Turnaround Time) is anywhere between few weeks to a month, which affects the campaign performance, leading to low conversions, poor lead quality & high marketing investments. Enterprise Software company in China specializing in commercial fleet telemetry. Use telemetry from buses to make them safer and more efficient to operate. Currently monitor 65K buses. Steady growth, was 10K buses on Jan 2013 Telemetry data to be stored in Trafodion tables for short latency access. Periodic ETL to Vertica for analysis Data ingest at about 10K – 30K rows per second. Concurrent access of data though relatively simple SELECT queries, at high concurrency, with sub-second response times OSS allows business users and partners to track status of orders and run scheduled and ad-hoc reports. This application has a need to move to Hadoop to save on S/W license cost. HP IceWall SSO is designed to adapt to various customers’ environments. It has a lot of parameters, templates and APIs in order to fulfill many kinds of customers’ requirements. And it is designed so that these customizations will not affect any future upgrades. HP IceWall SSO can flexibly connect using many types of authorization used in web applications. In particular it deals with 11 methods and 48 patterns of Form AuthN. Its latest version, HP IceWall SSO 10.0, now provides support for new leading-edge technologies such as cloud and virtualization, and the IceWall SSO product line has been extended to include Windows support in addition to the existing HP-UX and Linux versions.
  • 12. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12 + Open Source: How Trafodion is used?
  • 13. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13 Required Software Software Version HBase Version Linux 6.3,6.4 kernel Zookeeper V3.4.5 Trafodion 0.9.0 One of the following Hadoop Distribution: - Cloudera CDH 4.5 0.94.6 CDH 5.1.2 0.98 - Hortonworks HDP 1.3.3 0.94.6 HDP 2.1 0.98 - MapR M5 v3 0.94.13/0.94.17 Platforms Cloud VM Workstation Cluster Trafodion Installation Link - https://wiki.trafodion.org/wiki/index.php/Installation
  • 14. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14 Modern open source environment Source code in GitHub Build/test in OpenStack gerrit, zuul, jenkins Defect tracking in LaunchPad Documentation in MediaWiki Following best practices of OpenStack project
  • 15. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15 Building an Open Community Simple installation Meritocracy Share your expertise: Developing, fixing defects, testing, writing, translating and more Seeking early adopters Recruiting project contributors Discover our capabilities: Download and install in your Hadoop environment and take a test-drive www.trafodion.org Email: Generic Query- Project.Trafodion@hp.com Future directions – hema.ramaswamy@hp.com (Hema Ramaswamy, HP Labs)
  • 16. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16 + High Level Architecture
  • 17. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17 3-Layered High Level Architecture Client JDBC ODBC User and ISV Operational Applications Driver Hive Native Hive Tables Multi-Structured Data Store Integration HBase Native HBase Tables KVS, Columnar SQL ESP CMP Master ESPDTM WMS Compiler and Optimizer Workload Management(wip) SQL Parallelism Distributed Transaction Management . . . . Database Connectivity UDF Communicate with external processes HBase HDFS Relational Schema Trafodion Tables Storage Engines
  • 18. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18 Connectivity Architecture (T4) Client User and ISV Operational Applications ODBC/JDBC Drivers SQL DCS Master ZooKeeper Master Executor Master Executor Master Executor Master ExecutorMaster ExecutorMaster Executor DCS Server Database Connectivity Services . . . . Blue: control flow Green: data flow Connection Mgmt Process Mgmt DCS Server
  • 19. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19 Process Architecture Trafodion Node (DCS,EXE, ESP, CMP, DTM, UDF, WMS) VM or Physical Node Hadoop Data Node HBase APIs VM or Physical Node TCP/IP TCP/IP … Trafodion Metadata Trafodion Data HBase Region Server Hive/HDFS APIs Hive Data HDFS Data Trafodion Node (DCS,EXE, ESP, CMP, DTM, UDF, WMS) Hadoop Data Node HBase Region Server Hive/HDFS APIs HDFS Data Hive Data Trafodion Data Trafodion Metadata HBase APIs TCP/IP
  • 20. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20 Operational workload optimizations • Key-based access with SQL “pushdown” • Statistics based plan generation • Query plan caching • Data-flow, scheduler driven executor • DOP optimization • Parallelism without map-reduce • Secondary index support • Table structure optimizations Data-flow, Scheduler-driven Salting of Row Keys Optimized DOPStatistics Based Optimizations
  • 21. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21 + Demo
  • 22. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22 Enterprise-class SQL DBMS that uses native Hadoop formats (e.g., HBase, Hive, HDFS) for data storage Full-functioned ANSI SQL language support Standard ODBC/JDBC connectivity for Linux and Windows clients Low latency reads and writes via compile time and run time optimizations ACID distributed transaction protection over multiple stmts, tables, rows Support for big data sets using parallel SQL optimizations Retention of Hadoop benefits: reduced cost, scalability, elasticity and data redundancy Support for structured, unstructured, semi-structured data and flexible schemas Major Features - Recap
  • 23. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23 See for yourself… Come discover and develop on Trafodion www.trafodion.org
  • 24. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Thank You
  • 25. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Backup Slides
  • 26. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26 External Features
  • 27. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27 Trafodion DDL CREATE/DROP/ALTER statements • Tables, views, indexes, columns • Numeric, character, varchar, date, time, interval • Unicode (UTF8, UCS2) and single byte (ISO8859-1) for user data • UTF8 for metadata • Salting (Table partitioning for uniform data access across disks) LOBs • BLOB/CLOB datatypes (wip) Constraints • RI, Foreign Key, Primary Key, Check Security, Privileges • Grant/Revoke, create/drop user SPJ/UDF • Java Stored Procedures • User Defined Functions (wip)
  • 28. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.28 Trafodion DML Query statements • SELECT, INSERT, UPDATE, DELETE, UPSERT and MERGE Complex SQL operations • JOIN (INNER, LEFT/RIGHT/FULL OUTER), UNION, WHERE, GROUP BY, HAVING, ORDER BY, SAMPLING, TRANSPOSE, GROUP PIVOT, etc. Compile time and run time optimizations • Cost-based query optimizer, MDAM, OR optimizations • Correlated and nested subqueries Cursor support (non-holdable) SQL functions • Aggregate, date/time, character, mathematical, OLAP, sequence, etc.
  • 29. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.29 Miscellaneous Utilities • Update Statistics • Explain • Control Query Shape, Control Query Default Transaction Control • BEGIN WORK, COMMIT WORK, ROLLBACK WORK • SET TRANSACTION Oracle compatibility • Based on internal POCs • Syntax extensions (DUAL, ROWNUM, SYSDATE, NEXTVAL…) • Functions (TO_CHAR/DATE, SEQUENCE, Incompatible operations, …)
  • 30. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.30 Trafodion Interfaces Product interfaces • TrafCI • sqlci • Trafodion Query Workbench • ODBC client app • JDBC client app • T4 Driver • T2 Driver (wip) • HP DSM (wip) External Interfaces • DBVisualizer • SQuirrel
  • 31. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.31 Multiple Storage Engines Trafodion • Uses hbase tables underneath • native datatype storage format (ex: 2 bytes for a short) • Encoded data for serialization • column family and column name optimization Native Hbase • Cell format: One hbase cell per row output • Rowwise format: All cells in one row Hive • External metadata from hive • Text files: delimited data • Sequence files: structured data • ORC files: Optimized Row Columnar data (wip)
  • 32. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.32 Interoperability Trafodion, Hbase, Hive • Syntax extension to identify tables • Trafodion.sch.t_traf • Hbase.”_CELL_”.t_hbase • hive.hive.t_hive JOINs, INSERT…SELECT Transactional updates across storages • Currently Trafodion and HBase
  • 33. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.33 External integration • Table-valued UDF • Data Loader • Data Extractor • Other products through Table-valued UDF (wip) WMS (Workload Management) • Runtime stats data collection • Repository updates (wip) • Automated query control (wip) • Resource control (wip) • User control (wip)
  • 34. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.34 DISTRIBUT ED TRANSACTI ON MANAGEME NT (DTM)
  • 35. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.35 Overview Single row consistency using underlying HBase Global transactions across multiple tables/rows Optimistic locking • Conflict resolution at commit • First commit gets through HBase Trx EndPoint Coprocessors HBase Write Ahead Log (HLOG) for audit logging
  • 36. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.36 ... Node 1 SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library Transaction Manager HBase trx Region Server Node 2 SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library Transaction Manager HBase trx Region Server Node n SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library Transaction Manager HBase trx EndPoint Coprocessor Distributed, Scalable Architecture HBase trx EndPoint Coprocessor HBase trx EndPoint Coprocessor
  • 37. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.38 Performance Benchmarks
  • 38. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.39 Configuration for Perf Runs System under Test: Spinel ( Converted Gen8 Seaquest Cluster) Spinel is not an optimal configuration for Trafodion/Hbase due to the single raided data drive. Nodes: 10; Memory: 128 Gibper node; Cores: 16 per node; OS: RedhatLinux 6.3; Data Drives: 1 per node ( 3.4 TB Raid device) Hadoop/Hbase: Cloudera 4.5.0; Hadoop 2.0.0; Hbase 0.94.6/0.98 Trafodion Version 0.8.3
  • 39. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.40 YCSB (Yahoo Cloud Serving Benchmark) Workload A: read/update ratio 50/50 Traf 0.9.0
  • 40. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.41 DebitCredit 5 SQL statements per transaction (3 update, 1 insert, 1 select) Traf 0.9.0 Traf 0.9.0
  • 41. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.42 WORKLOA D OPTIMIZA TIONS
  • 42. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.43 Key-based access with SQL “pushdown” Statistics based plan generation Parallelism without map-reduce DOP(Degree Of Parallelism) optimization Adaptive segmentation Query plan caching Data-flow, scheduler driven executor Secondary index support Table structure optimizations
  • 43. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.44 Optimized DOPStatistics Based Optimizations
  • 44. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.45 Data-flow, Scheduler- driven Salting of Row Keys
  • 45. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.46 Real-Time Mixed Workload Concurrent transactional OpenCart application Queries against the same database Query type: OLTP Insert, Deletes, Selects Query monitoring
  • 46. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.47
  • 47. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.48 Operational Reporting Queries Complex multi-table operations Parallel query execution Various query optimizations for complex queries EXPLAIN: display query plan EXECUTE: run query
  • 48. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.49 Query: List of all products, some product info, current specials, a summary of their ratings and reviewsNested Join for keyed lookup into Trafodion Parallel scan larger Trafodion tables Cache of previous lookups into Trafodion
  • 49. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.50 Interoperability between Storages Access multiple Hadoop Storages Load/Extract data from Trafodion to/from Hive/HDFS Parallel insert and select operations Join tables from multiple storages Without external federation Storage access from within engine
  • 50. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.51 Load data from Trafodion tables to Hive table with insert-select statement Source data is detailed order information obtained by joining multiple Trafodion tables Parallel Join Trafodion tables acting as source Parallel insert into Hive Hive table is the target
  • 51. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.52 Transactional Capabilities Multiple rows Multiple tables Coordinated Commit and Rollback Conflict resolution
  • 52. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.53 TraFoto: Structured/Unstructured Integration Access structured relational Trafodion table Access unstructured native HBase table JOINs and other operations between the tables Transactional insert, update, deletes across both tables
  • 53. HP © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.54 UDFs: User Defined Functions User written scalar and tabular functions SQL syntax to access UDFs in queries Parallel execution at runtime

Editor's Notes

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 11
  11. 12
  12. 16
  13. 21
  14. Unlike most (if not all) NOSQL and other SQL-on-Hadoop products, Trafodion provides comprehensive ANSI SQL language support including full-functioned data definition (DDL), data manipulation (DML), transaction control (TCL) and database utility support. Unlike vanilla HBase, Trafodion provides support for creating and managing traditional relational database objects including tables, views, secondary indexes, and constraints. Columns (table attributes) are assigned data types as shown which are enforced by Trafodion. Internationalization (I18N) support is provided via Unicode encoding including UTF-8, UCS2, and ISO98859-1 for both user data as well as the database metadata. Comparisons and data manipulation between differing data encodings is transparently handled via implicit casting and translation support. Trafodion provides comprehensive and standard SQL data manipulation support including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with language options including join variants, unions, where predicates, aggregations (group by and having), sort ordering, sampling, correlated and nested sub-queries, cursors, and many SQL functions. Utilities are provided for updating table statistics used by the optimizer for costing (i.e. selectivity/cardinality estimates) plan alternatives, for displaying the chosen SQL execution plan, plan shaping, and a command line utility for interfacing with the database engine. Explicit control statements are provided to allow applications to define transaction boundaries and to abort transactions when warranted. Trafodion (post beta release) will support ANSI’s grant/revoke semantics to define user privileges in terms of managing and accessing the database objects.
  15. Unlike most (if not all) NOSQL and other SQL-on-Hadoop products, Trafodion provides comprehensive ANSI SQL language support including full-functioned data definition (DDL), data manipulation (DML), transaction control (TCL) and database utility support. Unlike vanilla HBase, Trafodion provides support for creating and managing traditional relational database objects including tables, views, secondary indexes, and constraints. Columns (table attributes) are assigned data types as shown which are enforced by Trafodion. Internationalization (I18N) support is provided via Unicode encoding including UTF-8, UCS2, and ISO98859-1 for both user data as well as the database metadata. Comparisons and data manipulation between differing data encodings is transparently handled via implicit casting and translation support. Trafodion provides comprehensive and standard SQL data manipulation support including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with language options including join variants, unions, where predicates, aggregations (group by and having), sort ordering, sampling, correlated and nested sub-queries, cursors, and many SQL functions. Utilities are provided for updating table statistics used by the optimizer for costing (i.e. selectivity/cardinality estimates) plan alternatives, for displaying the chosen SQL execution plan, plan shaping, and a command line utility for interfacing with the database engine. Explicit control statements are provided to allow applications to define transaction boundaries and to abort transactions when warranted. Trafodion (post beta release) will support ANSI’s grant/revoke semantics to define user privileges in terms of managing and accessing the database objects.
  16. Unlike most (if not all) NOSQL and other SQL-on-Hadoop products, Trafodion provides comprehensive ANSI SQL language support including full-functioned data definition (DDL), data manipulation (DML), transaction control (TCL) and database utility support. Unlike vanilla HBase, Trafodion provides support for creating and managing traditional relational database objects including tables, views, secondary indexes, and constraints. Columns (table attributes) are assigned data types as shown which are enforced by Trafodion. Internationalization (I18N) support is provided via Unicode encoding including UTF-8, UCS2, and ISO98859-1 for both user data as well as the database metadata. Comparisons and data manipulation between differing data encodings is transparently handled via implicit casting and translation support. Trafodion provides comprehensive and standard SQL data manipulation support including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with language options including join variants, unions, where predicates, aggregations (group by and having), sort ordering, sampling, correlated and nested sub-queries, cursors, and many SQL functions. Utilities are provided for updating table statistics used by the optimizer for costing (i.e. selectivity/cardinality estimates) plan alternatives, for displaying the chosen SQL execution plan, plan shaping, and a command line utility for interfacing with the database engine. Explicit control statements are provided to allow applications to define transaction boundaries and to abort transactions when warranted. Trafodion (post beta release) will support ANSI’s grant/revoke semantics to define user privileges in terms of managing and accessing the database objects.