Build Fast, Agile, Reliable Data Pipelines with Modern CDC

IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
6 Principles of Modern, Multi-Modal Change Data Capture
- How to Build Fast, Agile, Reliable Data Pipelines
John Santaferraro
Research Director
EMA
Nir Livneh
Founder and CEO
Equalum

Watch the On-Demand Webinar
Slide 2
 6 Principles of Modern, Multi-Modal Change Data Capture On-
Demand webinar: https://info.enterprisemanagement.com/six-
principles-of-modern-change-data-capture-webinar-ws
 Check out upcoming webinars from EMA here:
https://www.enterprisemanagement.com/freeResearch

Featured Speakers
Slide 3 © 2020 Enterprise Management Associates, Inc.
John Santaferraro, Research Director, EMA
John is the research director for analytics, business intelligence, and data management
at EMA. He has 23 years of experience in data and analytics, from startups to executive
positions at Fortune 50 companies. His deep understanding of the industry comes from
years of leadership in implementation, product and marketing organizations, along with
multiple big data imagineering efforts for finance, communications, retail, manufacturing,
healthcare, events, oil and gas, and utilities.
Nir Livneh, Founder and CEO, Equalum
Nir is a thought leader with over 20 years of experience in big data architecture and
performance. He led product management at Quest Software (Acquired by Dell) for all
big data products. Nir also led big data architecture projects at the Israeli Military
Intelligence unit (Equivalent to NSA). He is currently a member of the Forbes
Technology Council.

Agenda
4 © 2020 Enterprise Management Associates, Inc.
• Modernization Trends and the EMA Hybrid Data Ecosystem
• EMA Advisory: 6 Principles of Modern, Multi-Modal CDC
• EMA Guidance: Additional Checklist Items
• EMA Best Practices for Modern, Multi-Modal CDC Implementation
• Equalum Data Ingestion Platform
• Market Perspectives: The Advantages of Modern, Multi-Modal CDC

Trends Driving the Need for
Modern, Multi-Modal CDC

• Digital superiority is the new
mandate.
• Real time is the new
measurement.
• Intelligent automation is the
new means.
The Post-Pandemic Digital Shift

EMA Hybrid Data Ecosystem Architecture
• No one single platform for
all use cases
• New platforms added
based on value creation
• Interoperability is a must,
must play will with others

HDE Composition: Number of Platforms

Operating a Hybrid Data Ecosystem
• Orchestration defines the building of data pipelines to
acquire, prepare, provision, and adapt data for consistent
use across the entire data and application landscape.
• Operation is the delivery, administration, security,
availability, and automation of data movement among all
platforms and applications within the ecosystem.
• Optimization is the process of continually improving the
operation and orchestration of data pipelines.
• Obsolescence is the means by which legacy data
integrations and pipelines reach end of life and migrate to
new systems.

Consumer-Centric Design
The most important part
of the Hybrid Data
Ecosystem is outside
the technology circle:
the consumers.

EMA Advisory:
Six Principles of

• It is modern because it works in modern data ecosystems
• It is multi-modal because it supports…
• Real time and batch (fresh data, fast data)
• CDC and transformation
• Multi-structured data
• Multiple modes of replication
• Multi-directional data pipelines
• Hybrid ecosystems

Principle #1: Enable Comprehensive Data
Replication for CDC and Beyond
Streaming-First
Track data from
the point of the
event.
Rapid Replication
Replicate
changes as they
occur in near real
time.
Broad Replication
Initial data
capture
Schema migration
Grouping
Batch

Principle #2: Streamline Interconnectivity
Among Broad Set of Sources and Targets
• Sources become targets and targets become sources.
Technology Ecosystem Support
• Applications: SaaS and On-
Premises
• Data: Cloud and On-Premises
• Platforms: SQL, NoSQL,
Blockchain, others
Interconnectivity
Data Preparation
Bi-Directional

• Structured Data
• Multi-schema
• Semi Structured Data
• All File Formats
• Complex File Formats
o JSON, XML
• Mixed File Formats
o JSON in XML, XML in JSON
Standardization
Single Set of Operators
Easy to Combine
Common Ingestion
Principle #3: Standardize Support
for Multi-Structured Data

• Transformation
• Variables
• Aggregation
• Summarization
• Concatenation
• Correlations
• Others
Internet of Things:
Data collected by sensors often
includes irrelevant data and
requires enrichment before it is
analyzed. Data must be captured,
filtered, and enriched before it is
analyzed and elicits an intelligent
response.
Principle #4: Combine CDC and
Transformation for All Data Streams

• Orchestration should automate
the resolution of conflicts using
priority based on performance,
business criticality, and
dependencies.
• Optimization should continually
improve the operation and
orchestration of data pipelines.
Principle #5: Orchestrate and Optimize
Complex Stream-Based Operations
Abstraction Layer
Pipeline Complexity
Open Source Complexity

INDUSTRY ANALYSIS & CONSULTING18 © 2020 Enterprise Management Associates, Inc.
Principle #6: Create Interoperability in
Hybrid Environments
• 53% of all data is now in the cloud
• Interoperability includes CDC, replication, migration, and multi-
directional data pipelines.
On-premises Cloud SaaS Distributed Internet
Providers Apps of Things

EMA Guidance:
Additional Checklist Items for

Modern, Multi-Modal CDC Checklist
 Simple, easy to use
 Outperform legacy platforms
 Agile for changing data and business requirements
 Reliable to the point of unbreakable.
 Evolve with the latest open source projects

 Simple, easy to use
• Drag and drop
• No code
• Single pane of glass

 Outperform legacy platforms
• Performance speed
• Elasticity
• Time to Capture
• Time to Transform
• Time to Ingestion
• Time to Response

 Agile for changing data and business requirements
(Increase in data drift)
• Simple monitoring, versioning, etc.
• Auto-detection
• Auto-update
• Simple manual changes
• Exactly once in distributed systems

 Reliable to the point of unbreakable.
of participants are
spending between
30% and 50% of
their time fixing
breaks in data
pipelines.
61%

 Evolve with the latest open source projects

EMA Best Practices:
Implementation

Enrich and combine data streams for more
intelligence.
• Enrich and combine data streams for more intelligence.
• Standardize operators on data streams for dev speed.
• Audit data streams for greater accuracy.

INDUSTRY ANALYSIS & CONSULTINGSlide 28 © 2020 Enterprise Management Associates, Inc.
Data Ingestion Platform with
Infinite Scalability & Speed

INDUSTRY ANALYSIS & CONSULTING29
Equalum is an end-to-end streaming-first data ingestion platform Scalable
Fully Orchestrated
Easy to Deploy & Use
When choosing a
Data Ingestion Platform,
look for:
Performance & Reliability
Ease of Use
Broad Support for Sources &
Targets
Multi-Structured Data Support
Extensive ETL-Like Operations
OSS Foundation
Flexible Deployment

Market Perspectives:
The Advantages of

Modern, Multi-Modal CDC v. DIY
Modern, Multi-Modal CDC Do-It-Yourself
Time to Insight Hours Months
CDC Built In Integration Required
Broad Replication Built In Integration Required
Transformation Built In Integration Required
Source/Target Support Built In Separate Connectors
Multi-Structed Data Built In Separate Convertors
Hybrid Environments Built In Integration Required
Enrichment Built In Difficult to Build
Reliability Built In Difficult to Build
Agility Built In Difficult to Build
Ease of Use Built In Extremely Difficult
Validation Built In Integration Required
Orchestration Built In Difficult to Build
Optimization Built In Difficult to Build

Modern, Multi-Modal CDC v. Legacy CDC
and Integration
Modern, Multi-Modal CDC Legacy CDC, Integration
Time to Insight Hours Weeks
CDC Built In Legacy Only
Broad Replication Built In Legacy Only
Transformation Built In Built for Legacy
Source/Target Support Built In Mostly Legacy
Multi-Structed Data Built In Legacy Only
Hybrid Environments Built In Separate Products
Enrichment Built In Legacy Only
Reliability Built In Legacy Only
Agility Built In Legacy Only
Ease of Use Built In Legacy Only
Validation Built In Legacy Only
Orchestration Built In Legacy Only
Optimization Built In Legacy Only

Modern, Multi-Modal CDC v. Modern CDC
Modern, Multi-Modal CDC Modern CDC
Time to Insight Hours Limited
CDC Built In Little or No Legacy Support
Broad Replication Built In Little or No Legacy Support
Transformation Built In Little or No Legacy Support
Source/Target Support Built In Built In
Multi-Structed Data Built In Minimal Legacy Support
Hybrid Environments Built In Minimal Legacy Support
Enrichment Built In Minimal Legacy Support
Reliability Built In Minimal Legacy Support
Agility Built In Minimal Legacy Support
Ease of Use Built In Minimal Legacy Support
Validation Built In Minimal Legacy Support
Orchestration Built In Minimal Legacy Support
Optimization Built In Minimal Legacy Support

Equalum’s
Vision
for the
Future

Question and Answer: Log Questions in the Q&A panel
© 2020 Enterprise Management Associates, Inc.
Interested in Learning More?
LEARN ABOUT:
• Main Benefits of Using CDC
• 3 Common Approaches to CDC
• Disadvantages Enterprises Experience
When Relying On Legacy Tools
DOWNLOAD
www.equalum.io/modernize-wp

Build Fast, Agile, Reliable Data Pipelines with Modern CDC

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Build Fast, Agile, Reliable Data Pipelines with Modern CDC

Similar to Build Fast, Agile, Reliable Data Pipelines with Modern CDC (20)

More from Enterprise Management Associates

More from Enterprise Management Associates (20)

Recently uploaded

Recently uploaded (20)

Build Fast, Agile, Reliable Data Pipelines with Modern CDC