We are excited to announce the new general availability of the intuitive graphical interface for DataFunnel™. This browser-based point-and-click interface gives you the ability to move hundreds of relational tables to a different RSBMS – or to Hadoop – in just minutes! Select the schema of tables you’d like to move, filter out any tables, columns or rows you’d like to exclude, and invoke – all with the click of a mouse – in a user-friendly wizard interface.
View this webinar on-demand, where we discussed the newest features in Syncsort DMX/DMX-h, DMX CDC and DataFunnel™. During this webinar, you will see a special sneak peek of some of the new exciting additions coming soon to the Syncsort data integration product family! Webinar key takeaways:
• Learn about the newest features in the Syncsort Integrate product family
• Get a sneak preview of interesting Integrate features coming soon
• See the new intuitive independent DataFunnel™ platform interface
What’s New in Syncsort Integrate? New User Experience for Fast Data Onboarding
1. New Features Plus New
Onboarding User Experience
Paige Roberts, Integrate Product Marketing Manager
Ashwin Ramachandran, Integrate Product Manager
June 2018
1
2. New Features Plus New Onboarding User Experience
• Data Integration Product News
• DataFunnel UI Demo
• New Onboarding User Experience – DMX DataFunnel GUI
• Where to Find Out More
1 Hive Improvements
2 Impala Support
3 Lineage
4 DMX Change Data Capture
3. Disclaimer
Any information about our roadmap outlines our general
product direction and is subject to change at any time
without notice. It is for informational purposes only and
shall not, be incorporated into any contract or other
commitment.
Syncsort undertakes no obligation either to develop the
features or functionality described or to include any such
feature or functionality in a future release.
3
5. 5
Hive Support Enhancements
• JDBC connectivity
• Support for partitioned tables: ORC, Parquet, AVRO, HDFS
• Support for Truncate and Insert
• Automatic creation of Hive and other Hcat supported tables
• Direct distributed processing of Hive
• Update of Hive statistics
• Use Hive tables for lookups
• Change data capture target
• Hive ACID Merge support – Updates, inserts, deletes, and upserts in Hive
• Full support for complex types – arrays, structs, etc.
• Improved usability and support in mapping entire arrays, array elements and
composite fields.
6. 6
Cloudera Impala Support
• JDBC connectivity
• Both read and write support
• Support for Impala tables backed by: Parquet, Kudu
• Automatic creation of Impala tables
• Direct distributed processing of Impala – on edge node, framework-
designated node, or distributed on cluster with MapReduce or Spark
• Update of Impala statistics
• Use Impala tables for lookups
• Full support for update and insert
• Change data capture support
9. 9
Govern and Track Everything for Compliance
• Metadata and data lineage for Hive, Avro and Parquet
through HCatalog
• Metadata lineage export and REST API from DMX/DMX-h
o Simplify audits, analytics dashboards, metrics
o Integrate with enterprise metadata repositories
• Cloudera Navigator certified integration
o Audit and track data from source to cluster
o HDFS, YARN, Spark and other metadata
o Lineage, tagging
o Business and structural metadata
• Apache Atlas ingestion lineage integration
o Audit and track data from source to cluster
o Lineage, tagging
10. 10
Onboard ALL Enterprise Data – Mainframe to Streaming
Data Sources
Onboard data, modify
on-the-fly to match
Hadoop storage model,
or store unchanged for
archive and compliance.
Access data from
streaming and batch
sources outside
cluster.
Data Lake
Data
Transform, join,
cleanse, enhance
data in cluster
with MapReduce
or Spark.
Analytics,
Visualization,
Machine
Learning
Complete
Data
Analytics,
visualizations, and
machine learning
algorithms get ALL
necessary data.
11. 11
Get Source to Consumption, End-to-End Data Lineage
Data Sources
Auditors
get end-to-end
data lineage.
Analytics,
visualizations, and
machine learning
algorithms get ALL
necessary data.
Navigator or Atlas
gathers any other
changes made to
data on cluster.
Pass source-to-
cluster data
lineage info to
Navigator or Atlas.
Data Lake
Data changes made
by MapReduce,
Spark, HiveQL.
Data
Data Lineage
REST
API
Onboard data, modify
on-the-fly to match
Hadoop storage model,
or store unchanged for
archive and compliance.
Access data from
streaming and batch
sources outside
cluster.
Transform, join,
cleanse, enhance
data in cluster
with MapReduce
or Spark.
Complete
Data
Analytics,
Visualization,
Machine
Learning
15. Keep Your Data Fresh with Real-Time Change Data Capture
Keep data in sync in real-time
• Without overloading networks.
• Without affecting source database
performance.
• Without coding or tuning.
Reliable transfer of data you can trust even if connectivity fails on either side.
• Auto restart.
• No data loss.
Real-Time Replication
with Transformation
Conflict Resolution,
Collision Monitoring,
Tracking and Auditing
Files
RDBMS
Streams
Streams
RDBMS
Data
Lake
Mainframe
Cloud
OLAP
16. Real-Time Change Data Capture – New Sources and Targets
Sources
• IBM Db2 for z, i, LUW
• Apache Kafka
• VSAM
• IBM Informix
• MS SQL Server
• Oracle
• Oracle RAC
• Sybase
Targets
• Hive
• HDFS
• Apache Kafka
• Impala
• IBM Db2
• MS Azure SQL
• SQL Server
• PostgreSQL
• MySQL
• Oracle
• Oracle RAC
• Sybase
• Teradata
• Apache Kafka - Streaming
• Amazon Kinesis
17. 17
Onboard Relational Data Quickly
• Extract, map and move whole DB schemas in one invocation
• Extract from Oracle, Db2, MS SQL Server, Teradata, Netezza and Redshift
• To SQL Server, Postgres, Hive, HDFS, Redshift and Amazon S3
• Automatically create target Hive and HCat tables
• Onboard hundreds of tables into your cluster
• Onboard whole database schemas at once
• Create target tables automatically in Hive
• Transform data in flight
• Filter unwanted:
o Tables
o Rows
o data types
o columns
DMX DataFunnel™
19. 19
THANK YOU!
Check out the DMX DataFunnel New User Experience Demo!
http://www.syncsort.com/en/Resource-Center/BigData/Videos/DMX-DataFunnel-New-User-Interface-Makes-Database