Trillium Software System:
Enterprise
Paige Roberts, Product Marketing Manager
Steve Shissler, Director, Sales Engineering
1
Housekeeping
Webcast Audio
• Today’s webcast audio is streamed through your computer speakers.
• If you need technical assistance with the web interface or audio,
please reach out to us using the chat window.
Questions Welcome
• Submit your questions at any time during the presentation
using the chat window.
• We will answer them during our Q&A session following the
presentation.
Recording and slides
• This webcast is being recorded. You will receive an
email following the webcast with a link to download
both the recording and the slides.
2
Agenda
1 Syncsort
2 New in Syncsort Data Quality
3 Performance Increases
4 TSS Enterprise - Integrated Real-Time Data Quality
5 Application Integration Example – Collibra
6 Questions
Who is
Syncsort?
>7,000 customers
84 of the Fortune 100
Customers in >100 countries
Headquarters: Pearl River, NY
U . S . L O C AT I O N S
• Burlington, MA; Irvine, CA;
Oakbrook Terrace, IL; Rochester, MN
G L O B A L P R E S E N C E
• U.K., France, Germany, Netherlands,
Israel, Hong Kong & Japan
Big Iron to Big Data is a fast-growing
market segment composed of solutions
that optimize traditional data systems
and deliver mission-critical data from
these systems to next-generation
analytic environments.
4
Global leader in
Big Iron to Big Data
Syncsort’s Trillium Software System:
What’s New?
Trillium Quality for Big Data
Trillium Quality =
Best-of-breed data quality
solution.
Leader in Gartner Data
Quality Tools MQ 12 years
running.
Intelligent Execution =
Artificially intelligent
dynamic performance
optimizer for cluster
execution in MapReduce,
Amazon EMR, or Spark.
6
Trillium Quality +
Intelligent Execution =
High performance
industry-leading data
quality on Big Data and
Cloud platforms.
• Build data quality processes that
ensure high-quality data that
meets such key business needs as:
o Single customer view (SCV)
o Standardized product data
o Standardization for fraud detection
7
Trillium Quality – Powerful Data Cleansing
• Consolidate data sources on input
• Match on party, household, business, etc.
• Develop workflows to transform, parse,
standardize, match and survive best record
• Manage “householding” issues associated with
multiple physical addresses under a single account
KEY FUNCTIONALITY:
• Global address validation with individual country postal rules
• Enrich missing postal information, latitude/longitude and other reference data
8
Design Once, Deploy Anywhere
Intelligent Execution - Insulate your organization from underlying complexities of Hadoop.
Get excellent performance every time
without tuning, load balancing, etc.
No re-design, re-compile, no re-work ever
• Future-proof job designs for emerging
compute frameworks, e.g. Spark 2.x
• Move from dev to test to production
• Move from on-premise to Cloud
• Move from one Cloud to another
Use existing ETL skills
No parallel programming – Java, MapReduce, Spark …
No worries about:
• Mappers, Reducers
• Big side or small side of joins …
Design Once
in visual GUI
Deploy Anywhere!
On-Premise,
Cloud
Mapreduce, Spark,
Future Platforms
Windows, Unix,
Linux
Batch,
Streaming
Single Node,
Cluster
9
Trillium Quality for Big Data
• Deploy data quality workflows as native, parallel MapReduce or Spark
processes for optimal efficiency.
• Process hundreds of millions of records of data.
• Standardize, enhance, and match international data sets with postal and
country-code validation.
• Integrate, parse, standardize, and match new and legacy customer data
from multiple disparate sources.
• Increase processing efficiency.
• Support failover through Hadoop’s fault-tolerant design; during a node
failure, processing is redirected to another node.
10
Two Ways to Get Postal Updates
Trillium Postal Download Web Service
Trillium Postal Download Web Service is an
automated download service introduced in
TSS v15.7. The download service allows you
to check the status of your postal license and
download the postal directories from a
browser-based application.
TSS Download Center (File Portal) FTP website
TSS Download Center allows you to manually download
postal directories through Trillium Software’s secure
website. See the Trillium Software System Installation
Guide for procedures on downloading postal directories
through this website.
11
Trillium Discovery Documented REST APIs
Trillium Discovery REST APIs installed with TSS server,
documentation in Help file for easy integration with
other applications like ASG Data Intelligence, Collibra,
etc.
12
Collibra Integration
Collibra can define and manage data quality
rules, but cannot enforce the rules on the
data or measure compliance to them.
Goal:
• Make data accessible, traceable and
meaningful to business users.
• Automatically, pass Collibra rules into Trillium
Discovery and get rule compliance data passed
back to Collibra
Requirements:
• Bi-directional near real-time integration
between Trillium Discovery and Collibra DGC
for quality measurement and monitoring
• Trillium business rule analysis results / data
quality metrics shown in Collibra dashboards.
• Data Stewards can quickly identify issues and
take corrective action when data quality
standards are not met.
Closing the Loop
Collibra Data Governance Center
• Enables non-technical users to define
business policies and data quality rules
in plain language
• Makes data quality performance
available to all users
Trillium Discovery
• Imports DBC business rules so technical user
can convert to executable data quality rules
• Constantly runs data quality metrics on near
real-time basis, passes results back to
Collibra dashboards
Rulebooks to Rules
Quality test Results
Bi-directional connectivity Constant sync
Metric falling below
thresholds can
trigger case in
Collibra Issue
Management
13
14
And more …
• Unique ID (UUID) Function
• Trillium Language Pack Locale Setting
• Apache Tomcat Upgrade to v8.5.32
• And more …
Example:
German locale setting in config.txt
key rest_api {
value locale "de"
}
Syncsort’s Trillium Software System
Performance Improvements
Trillium Software
System® at FedEx
“The Trillium Software System
lets FedEx target specific data
issues and quickly modify rules
to resolve them.”
— senior technical analyst
Accurately link and match
shipper and receiver data to
add value to FedEx InSight
customer service portal
• Inaccurate and inconsistent
customer name and address
data
• Free-form text with large
amounts of non-address data
• High transaction volumes – up
to 500,000/hour
S O L U T I O N
• Trillium Software System
• Accurate record
matches in under a
secondusing all available
information, including free-
form text
• Implemented in < 4 months16
• Competitive edge – Drives
higher customer usage
• High customer loyalty –
even when competitor has
lower price
• Easy global expansion –
proven in US and Canada,
now global
17
Trillium Quality Performance Improvement
0
100
200
300
400
500
600
700
Windows Linux
Records/Second
Performance Comparison
15.7.4 15.8
Trillium Discovery
at Babcock Marine
& Technology
“Data is the DNA of our supply
chain. Data completeness and
accuracy are critical to our current
and future operations in an
increasingly competitive market.”
— Andy Chapell
Head of Supply Chain Capability
Need to grow the business globally
and secure competitive edge
• > 1 million parts and commodities
from > 3000 suppliers
• Complex data
• Disparate systems
S O LU T I O N
Trillium Discovery
• Supply chain users assess
completeness, accuracy, review
failing data rows and export
• Resolve inconsistencies quickly
and accurately
• Less rework = less cost
• Better reporting, analysis =
Better decision-making
• Improved compliance
18
23% improvement
in supplier master data quality
4 of 11 business units hit
hit 98% data quality goal
6 more business units
exceeded 95% data quality
19
Trillium Discovery Performance Improvement
0
2,000
4,000
6,000
8,000
10,000
12,000
100k 500k 1 Million 5 Million 10 Million 20 Million 50 Million 93.8 Million
Time(seconds)
Number of Records Processed
Load and Profile Data (13 Attributes)
15.8.0 (64-bit)
15.7.4 (32-bit)
20
Trillium Discovery Performance Improvement
0
200
400
600
800
1000
1200
1400
1600
1800
100k 500K 1 Million 2 Million 3 Million 5 Million 7 Million 9.9 Million
Time(seconds)
Number of Records
Load and Profile Data (176 Attributes)
15.8.0 (64-bit)
15.7.4 (32-bit)
21
Trillium Quality for Big Data Performance Improvement
0
500
1000
1500
2000
2500
MapReduce (Fixed) Spark (Fixed) MapReduce (Delim) Spark (Delim)
Records/Second
Performance Comparison
15.7.4 15.8
Syncsort’s TSS Enterprise
Integrated Real-Time Data Quality
23
TSS Enterprise Architecture
Trillium Cleanser
Web Service
Global Address
Verification Files
HTTPS
Trillium Matcher
Web Service
Deployed
Project
Web Key
User Key
Project
Name
Trillium Control Center
Or
REST or SOAP
Call
Your
Application
• Enter Contacts or Leads as normal
• Pop-up validation of Address
(optionally email & phone)
Apply Data Quality to Data Entry in Real-Time
Trillium Quality for Microsoft Dynamics CRM
can validate new data as it is entered:
24
• Match to existing Contacts and Leads
(cross-entity) with cross-population
of validated fields
• Option to merge between Leads and
Contacts
• Choose which fields to be merged
into the surviving master record
Apply Data Quality to Data Entry in Real-Time
Trillium Quality for Microsoft Dynamics CRM
can deduplicate new data as it is entered:
25
Trillium Software
System® Enterprise
at Porsche
“With data quality being
managed successfully, nothing
now stands in the way of the
Porsche CRM.”
- CRM Project Manager, PORSCHE
Need to enhance customer
relationships around the world
• Localized needs in a global company
• Data from disparate systems
• Combine multiple records into a
single customer view
• mySAP certification
S O L U T I O N
• Trillium Software System
standardizes, cleanses and validates
data, with language and format
localization, 10 - 15X faster than
SAP certification
requirements
• Flexible matching rules merge
non-identical duplicate customer
records
• Trillium professional services26
• Improved worldwide dealer
support
• Enhanced customer
communications, customer
satisfaction
• Increased brand loyalty
• More targeted, relevant
marketing
Syncsort’s TSS Enterprise
Application Integration
Collibra Integration Example
28
Trillium Software Connection to Collibra
Connect (ESB) Server
Gateway
Connect
ApplicationPORT 443
TLS
Basic Auth with LDAP for SSO
Connect
Application
29
Business Rule created in Collibra
30
Same Rule ‘pushed’ to Trillium
31
Rule detail in Trillium before created by Data Steward
32
Rule Detail in Trillium after created by Data Steward
33
Rule results in Trillium
34
Updated Rulebook in Collibra after ‘Push back’
Data Quality Metric was created for rule results
35
Data Quality Rule and Metric with data from Trillium
Predicate = Expression from Trillium
36
Rules and Data Quality Metrics in Traceability Diagram
Questions?
38
What's New in Syncsort's Trillium Line of Data Quality Software - TSS Enterprise!

What's New in Syncsort's Trillium Line of Data Quality Software - TSS Enterprise!

  • 1.
    Trillium Software System: Enterprise PaigeRoberts, Product Marketing Manager Steve Shissler, Director, Sales Engineering 1
  • 2.
    Housekeeping Webcast Audio • Today’swebcast audio is streamed through your computer speakers. • If you need technical assistance with the web interface or audio, please reach out to us using the chat window. Questions Welcome • Submit your questions at any time during the presentation using the chat window. • We will answer them during our Q&A session following the presentation. Recording and slides • This webcast is being recorded. You will receive an email following the webcast with a link to download both the recording and the slides. 2
  • 3.
    Agenda 1 Syncsort 2 Newin Syncsort Data Quality 3 Performance Increases 4 TSS Enterprise - Integrated Real-Time Data Quality 5 Application Integration Example – Collibra 6 Questions
  • 4.
    Who is Syncsort? >7,000 customers 84of the Fortune 100 Customers in >100 countries Headquarters: Pearl River, NY U . S . L O C AT I O N S • Burlington, MA; Irvine, CA; Oakbrook Terrace, IL; Rochester, MN G L O B A L P R E S E N C E • U.K., France, Germany, Netherlands, Israel, Hong Kong & Japan Big Iron to Big Data is a fast-growing market segment composed of solutions that optimize traditional data systems and deliver mission-critical data from these systems to next-generation analytic environments. 4 Global leader in Big Iron to Big Data
  • 5.
    Syncsort’s Trillium SoftwareSystem: What’s New?
  • 6.
    Trillium Quality forBig Data Trillium Quality = Best-of-breed data quality solution. Leader in Gartner Data Quality Tools MQ 12 years running. Intelligent Execution = Artificially intelligent dynamic performance optimizer for cluster execution in MapReduce, Amazon EMR, or Spark. 6 Trillium Quality + Intelligent Execution = High performance industry-leading data quality on Big Data and Cloud platforms.
  • 7.
    • Build dataquality processes that ensure high-quality data that meets such key business needs as: o Single customer view (SCV) o Standardized product data o Standardization for fraud detection 7 Trillium Quality – Powerful Data Cleansing • Consolidate data sources on input • Match on party, household, business, etc. • Develop workflows to transform, parse, standardize, match and survive best record • Manage “householding” issues associated with multiple physical addresses under a single account KEY FUNCTIONALITY: • Global address validation with individual country postal rules • Enrich missing postal information, latitude/longitude and other reference data
  • 8.
    8 Design Once, DeployAnywhere Intelligent Execution - Insulate your organization from underlying complexities of Hadoop. Get excellent performance every time without tuning, load balancing, etc. No re-design, re-compile, no re-work ever • Future-proof job designs for emerging compute frameworks, e.g. Spark 2.x • Move from dev to test to production • Move from on-premise to Cloud • Move from one Cloud to another Use existing ETL skills No parallel programming – Java, MapReduce, Spark … No worries about: • Mappers, Reducers • Big side or small side of joins … Design Once in visual GUI Deploy Anywhere! On-Premise, Cloud Mapreduce, Spark, Future Platforms Windows, Unix, Linux Batch, Streaming Single Node, Cluster
  • 9.
    9 Trillium Quality forBig Data • Deploy data quality workflows as native, parallel MapReduce or Spark processes for optimal efficiency. • Process hundreds of millions of records of data. • Standardize, enhance, and match international data sets with postal and country-code validation. • Integrate, parse, standardize, and match new and legacy customer data from multiple disparate sources. • Increase processing efficiency. • Support failover through Hadoop’s fault-tolerant design; during a node failure, processing is redirected to another node.
  • 10.
    10 Two Ways toGet Postal Updates Trillium Postal Download Web Service Trillium Postal Download Web Service is an automated download service introduced in TSS v15.7. The download service allows you to check the status of your postal license and download the postal directories from a browser-based application. TSS Download Center (File Portal) FTP website TSS Download Center allows you to manually download postal directories through Trillium Software’s secure website. See the Trillium Software System Installation Guide for procedures on downloading postal directories through this website.
  • 11.
    11 Trillium Discovery DocumentedREST APIs Trillium Discovery REST APIs installed with TSS server, documentation in Help file for easy integration with other applications like ASG Data Intelligence, Collibra, etc.
  • 12.
    12 Collibra Integration Collibra candefine and manage data quality rules, but cannot enforce the rules on the data or measure compliance to them. Goal: • Make data accessible, traceable and meaningful to business users. • Automatically, pass Collibra rules into Trillium Discovery and get rule compliance data passed back to Collibra Requirements: • Bi-directional near real-time integration between Trillium Discovery and Collibra DGC for quality measurement and monitoring • Trillium business rule analysis results / data quality metrics shown in Collibra dashboards. • Data Stewards can quickly identify issues and take corrective action when data quality standards are not met.
  • 13.
    Closing the Loop CollibraData Governance Center • Enables non-technical users to define business policies and data quality rules in plain language • Makes data quality performance available to all users Trillium Discovery • Imports DBC business rules so technical user can convert to executable data quality rules • Constantly runs data quality metrics on near real-time basis, passes results back to Collibra dashboards Rulebooks to Rules Quality test Results Bi-directional connectivity Constant sync Metric falling below thresholds can trigger case in Collibra Issue Management 13
  • 14.
    14 And more … •Unique ID (UUID) Function • Trillium Language Pack Locale Setting • Apache Tomcat Upgrade to v8.5.32 • And more … Example: German locale setting in config.txt key rest_api { value locale "de" }
  • 15.
    Syncsort’s Trillium SoftwareSystem Performance Improvements
  • 16.
    Trillium Software System® atFedEx “The Trillium Software System lets FedEx target specific data issues and quickly modify rules to resolve them.” — senior technical analyst Accurately link and match shipper and receiver data to add value to FedEx InSight customer service portal • Inaccurate and inconsistent customer name and address data • Free-form text with large amounts of non-address data • High transaction volumes – up to 500,000/hour S O L U T I O N • Trillium Software System • Accurate record matches in under a secondusing all available information, including free- form text • Implemented in < 4 months16 • Competitive edge – Drives higher customer usage • High customer loyalty – even when competitor has lower price • Easy global expansion – proven in US and Canada, now global
  • 17.
    17 Trillium Quality PerformanceImprovement 0 100 200 300 400 500 600 700 Windows Linux Records/Second Performance Comparison 15.7.4 15.8
  • 18.
    Trillium Discovery at BabcockMarine & Technology “Data is the DNA of our supply chain. Data completeness and accuracy are critical to our current and future operations in an increasingly competitive market.” — Andy Chapell Head of Supply Chain Capability Need to grow the business globally and secure competitive edge • > 1 million parts and commodities from > 3000 suppliers • Complex data • Disparate systems S O LU T I O N Trillium Discovery • Supply chain users assess completeness, accuracy, review failing data rows and export • Resolve inconsistencies quickly and accurately • Less rework = less cost • Better reporting, analysis = Better decision-making • Improved compliance 18 23% improvement in supplier master data quality 4 of 11 business units hit hit 98% data quality goal 6 more business units exceeded 95% data quality
  • 19.
    19 Trillium Discovery PerformanceImprovement 0 2,000 4,000 6,000 8,000 10,000 12,000 100k 500k 1 Million 5 Million 10 Million 20 Million 50 Million 93.8 Million Time(seconds) Number of Records Processed Load and Profile Data (13 Attributes) 15.8.0 (64-bit) 15.7.4 (32-bit)
  • 20.
    20 Trillium Discovery PerformanceImprovement 0 200 400 600 800 1000 1200 1400 1600 1800 100k 500K 1 Million 2 Million 3 Million 5 Million 7 Million 9.9 Million Time(seconds) Number of Records Load and Profile Data (176 Attributes) 15.8.0 (64-bit) 15.7.4 (32-bit)
  • 21.
    21 Trillium Quality forBig Data Performance Improvement 0 500 1000 1500 2000 2500 MapReduce (Fixed) Spark (Fixed) MapReduce (Delim) Spark (Delim) Records/Second Performance Comparison 15.7.4 15.8
  • 22.
  • 23.
    23 TSS Enterprise Architecture TrilliumCleanser Web Service Global Address Verification Files HTTPS Trillium Matcher Web Service Deployed Project Web Key User Key Project Name Trillium Control Center Or REST or SOAP Call Your Application
  • 24.
    • Enter Contactsor Leads as normal • Pop-up validation of Address (optionally email & phone) Apply Data Quality to Data Entry in Real-Time Trillium Quality for Microsoft Dynamics CRM can validate new data as it is entered: 24
  • 25.
    • Match toexisting Contacts and Leads (cross-entity) with cross-population of validated fields • Option to merge between Leads and Contacts • Choose which fields to be merged into the surviving master record Apply Data Quality to Data Entry in Real-Time Trillium Quality for Microsoft Dynamics CRM can deduplicate new data as it is entered: 25
  • 26.
    Trillium Software System® Enterprise atPorsche “With data quality being managed successfully, nothing now stands in the way of the Porsche CRM.” - CRM Project Manager, PORSCHE Need to enhance customer relationships around the world • Localized needs in a global company • Data from disparate systems • Combine multiple records into a single customer view • mySAP certification S O L U T I O N • Trillium Software System standardizes, cleanses and validates data, with language and format localization, 10 - 15X faster than SAP certification requirements • Flexible matching rules merge non-identical duplicate customer records • Trillium professional services26 • Improved worldwide dealer support • Enhanced customer communications, customer satisfaction • Increased brand loyalty • More targeted, relevant marketing
  • 27.
    Syncsort’s TSS Enterprise ApplicationIntegration Collibra Integration Example
  • 28.
    28 Trillium Software Connectionto Collibra Connect (ESB) Server Gateway Connect ApplicationPORT 443 TLS Basic Auth with LDAP for SSO Connect Application
  • 29.
  • 30.
  • 31.
    31 Rule detail inTrillium before created by Data Steward
  • 32.
    32 Rule Detail inTrillium after created by Data Steward
  • 33.
  • 34.
    34 Updated Rulebook inCollibra after ‘Push back’ Data Quality Metric was created for rule results
  • 35.
    35 Data Quality Ruleand Metric with data from Trillium Predicate = Expression from Trillium
  • 36.
    36 Rules and DataQuality Metrics in Traceability Diagram
  • 37.
  • 38.