Digital transformation is producing massive new data and pushing data consumption requirements to real time. The result is a race among vendors to shift from data integration to interoperability. These slides based on webinar from leading IT research firm EMA provide insights into new EMA research that characterizes modern data practices and identifies best practices for your modernization journey.

1. John Santaferraro
Research Director, EMA
jsantaferraro@emausa.com
@santaferraro
Modernization: Moving From Data
Integration to Interoperability

2. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING2 © 2019 Enterprise Management Associates
Watch the On-Demand Webinar
• Modernization: Moving From Data Integration to
Interoperability On-Demand webinar:
https://ema.wistia.com/medias/nclrdv0h9w#
• Check out upcoming webinars from EMA here:
http://www.enterprisemanagement.com/freeResearch

3. Featured Speaker
Slide 3 © 2019Enterprise Management Associates, Inc.
John Santaferraro, Research Director, EMA
John is the research director for analytics, business
intelligence, and data management at EMA. He has
23 years of experience in data and analytics, from
startups to executive positions at Fortune 50
companies. His deep understanding of the industry
comes from years of leadership in implementation,
product and marketing organizations, along with
multiple big data imagineering efforts for finance,
communications, retail, manufacturing, healthcare,
events, oil and gas, and utilities.

4. Sponsor
Infoworks offers the most comprehensive and automated Enterprise Data Operations
and Orchestration (EDO2) system. It is the only EDO2 system built to automate and
accelerate deployment and orchestration of analytics projects at scale, in cloud, hybrid,
multi-cloud, and premise-based environments. Through deep automation and a code-
free environment, Infoworks empowers organizations to rapidly consolidate and
organize enterprise data, create analytics pipelines and deploy projects to production
within days – dramatically increasing business agility and accelerating time-to-value.
Infoworks counts some of the world’s largest financial, retail, technology, healthcare, oil
& gas, and manufacturing companies as its customers. Founded in 2014 and
headquartered in Palo Alto, the company is funded by NEA, Nexus Venture Partners,
Centerview Capital Technology, and Knoll Ventures. To learn more, please visit
infoworks.io.
Slide 4 © 2019 Enterprise Management Associates, Inc.

5. Agenda
• What is Modern Data Integration (MDI)?
• What is Driving the Modernization of Data Integration?
• The MDI State of the Union
• The Challenges of MDI
• Modernization Best Practices
- Automation
- Pipeline Management
Slide 5 © 2019 Enterprise Management Associates, Inc.

6. Slide 6 © 2019Enterprise Management Associates, Inc.
What is Modern Data Integration?

7. The History of Data Integration
Slide 7 © 2019Enterprise Management Associates, Inc.
1980’s COBOL
1990’s Simple Integration
Separate Integration
2000’s Point to Point
Big Data Integration
iPaaS
2010’s Integration Platforms
Streaming Platforms

8. What is Modern Data Integration
Modern Data Integration (MDI) is the modernization of
legacy data integration. MDI consolidates integration
into multi-purpose platforms, connects entire
ecosystems, and drives interoperability. Through the
use of artificial intelligence and machine learning,
along with rich metadata, these new technologies
usher in a new era of self-service, hyper-speed, and
the simplified management of complex,
multidirectional data pipelines.
Slide 8 © 2019Enterprise Management Associates, Inc.

9. EMA Point of View on MDI
Slide 9 © 2019 Enterprise Management Associates, Inc.
Companies that invest in modern data integration
platforms will experience an acceleration of insight-
driven decisions, extend the reach of their analytics
programs and gain a competitive advantage.
Vendors that excel in the use of machine learning
for the autonomous development, deployment, and
orchestration of complex data pipelines in
their modern data integration platforms will create
significant differentiation and barriers to entry.

10. Slide 10 © 2019Enterprise Management Associates, Inc.
What is Driving the Modernization of
Data Integration?

11. What are the business drivers for the modernization of
your organization's overall data integration program?
Slide 11 © 2019 Enterprise Management Associates, Inc.

12. What are the technical drivers for the modernization of
your organization's overall data integration program?
Slide 12 © 2019 Enterprise Management Associates, Inc.

13. What are the key drivers behind the modernization of
your data integration, preparation, and pipeline
management platforms?
Slide 13 © 2019 Enterprise Management Associates, Inc.

14. What is driving your organization's need for
more real-time integration?
Slide 14 © 2019 Enterprise Management Associates, Inc.

15. What is driving your organization's need for
cloud integration?
Slide 15 © 2019 Enterprise Management Associates, Inc.

16. Slide 16 © 2019Enterprise Management Associates, Inc.
The MDI State of the Union

17. What distribution of data types flows through
your organization's data integration program?
Slide 17 © 2019 Enterprise Management Associates, Inc.

18. What percentage of your organization's data is
stored in the cloud versus on-premises?
Slide 18 © 2019 Enterprise Management Associates, Inc.

19. What is the distribution of your organization's integration
work across cloud, iPaaS, on-premises, or desktop?
Slide 19 © 2019 Enterprise Management Associates, Inc.

20. What percentage of your organization's integration
work is done with the following tools or processes?
Slide 20 © 2019 Enterprise Management Associates, Inc.
25%
18%
15%
13%
13%
11%
9%
6%
5%
0% 20% 40%
Data integration or preparation platform
API or application integration platform
Push down to database or big data platform
Manual coding
Pipeline management or automation platform
Data warehouse automation platform
Streaming platform
Messaging platform
Ecosystem integration platform

21. What percentage of your organization's integration
work is done with the following tools or processes?
Slide 21 © 2019 Enterprise Management Associates, Inc.

22. What is the distribution of data sources flowing through
your organization's overall integration program?
Slide 22 © 2019 Enterprise Management Associates, Inc.

23. What is the distribution of your organization's integration
processing through server technology versus through
serverless or agent technology?
Slide 23 © 2019 Enterprise Management Associates, Inc.

24. On average, how long does it take to develop
a data pipeline?
Slide 24 © 2019 Enterprise Management Associates, Inc.
3%
9%
11%
13%
10%
14%
12%
16%
6%
5%
0% 5% 10% 15% 20%
Under 15 minutes
15 to 60 minutes
1-3 hours
4-10 hours
11-24 hours
1-2 days
3-5 days
1-3 weeks
1-2 months
2+ months

25. On average, how long does it take to make a
data pipeline operational in production?
Slide 25 © 2019 Enterprise Management Associates, Inc.
2%
6%
12%
16%
5%
15%
12%
16%
10%
7%
0% 5% 10% 15% 20%
Under 15 minutes
15 to 60 minutes
1-3 hours
4-10 hours
11-24 hours
1-2 days
3-5 days
1-3 weeks
1-2 months
2+ months

26. Comparison of Pipeline Development Time
and Deployment Time
Slide 26 © 2019 Enterprise Management Associates, Inc.
3%
9%
11%
13%
10%
14%
12%
16%
6%
5%
2%
6%
12%
16%
5%
15%
12%
16%
10%
7%
0% 2% 4% 6% 8% 10% 12% 14% 16% 18%
Under 15 minutes
15 to 60 minutes
1-3 hours
4-10 hours
11-24 hours
1-2 days
3-5 days
1-3 weeks
1-2 months
2+ months
Development Deployment

27. On average, what percentage of your time do you spend
managing break and fix issues for data pipelines?
Slide 27 © 2019 Enterprise Management Associates, Inc.

28. On average, what is your mean time to repair
(MTTR) for breaks in data pipelines?
Slide 28 © 2019 Enterprise Management Associates, Inc.
2%
12%
24%
23%
20%
10%
6%
2%
0%
0% 5% 10% 15% 20% 25% 30%
Under 15 minutes
15 to 60 minutes
1-3 hours
4-10 hours
11-24 hours
1-2 days
3-5 days
1-3 weeks
1-2 months

29. How do you currently manage conflicts among
multiple data pipelines?
Slide 29 © 2019 Enterprise Management Associates, Inc.
20%
20%
17%
17%
8%
7%
5%
5%
0% 5% 10% 15% 20% 25%
We manage conflicts manually
We manage conflicts with a data
integration platform
We manage conflicts with a job
scheduler
We manage conflicts with hand-coded
scripts
We manage conflicts with a pipeline
automation platform
We manage conflicts with a big data
platform
We don't manage conflicts
We manage conflicts with a pipeline
management platform

30. How do you plan to manage conflicts among
multiple data pipelines in the future?
Slide 30 © 2019 Enterprise Management Associates, Inc.
17%
20%
8%
20%
7%
17%
5%
5%
21%
21%
20%
13%
12%
9%
8%
5%
0% 10% 20% 30%
We plan to manage conflicts with a job
scheduler
We plan to manage conflicts with a data
integration platform
We plan to manage conflicts with a pipeline
automation platform
We plan to manage conflicts manually
We plan to manage conflicts with a big data
platform
We plan to manage conflicts with hand-
coded scripts
We plan to manage conflicts with a pipeline
management platform
We don't have any plans to manage conflicts
Current Future

31. What percentage of your codebase is portable
and reusable for future migrations?
Slide 31 © 2019 Enterprise Management Associates, Inc.
2%
8%
19%
23%
12%
17%
13%
3%
2%
0% 5% 10% 15% 20% 25%
10%
20%
30%
40%
50%
60%
70%
80%
90%

32. What percentage of your codebase is portable
and reusable for future migrations?
Slide 32 © 2019 Enterprise Management Associates, Inc.

33. Slide 33 © 2019Enterprise Management Associates, Inc.
The Challenges of Modern
Data Integration

34. What are the biggest challenges for your
organization's data integration program?
Slide 34 © 2019 Enterprise Management Associates, Inc.
36%
30%
28%
27%
27%
27%
27%
21%
20%
20%
11%
7%
0% 20% 40%
Lack of skilled resources
Too many changes in the data
Too much time spent on break and fix
Integration tools that don't scale
Not enough resources
Too many changes in the business requirements
Too much complexity
Too many data cleansing issues
Too much hand coding
Not enough server resources
Too many different integration tools
The business doesn't understand what we do

35. What are the biggest challenges you face in
doing real-time data integration?
Slide 35 © 2019 Enterprise Management Associates, Inc.
30%
28%
28%
28%
24%
22%
21%
20%
18%
13%
0% 20% 40%
Analysis of data is too slow or difficult
Validation of data is too slow or difficult
Preparation of data is too slow or difficult
Lack of skilled resources for real-time integration
Synchronization of data is too slow or difficult
Storage of the data is too slow or difficult
Lack of tools for real-time integration
Delivery of the insight is too slow or difficult
Acquisition of data is too slow or difficult
We don't do real-time data integration

36. What are the biggest challenges you are facing with
cloud-based data integration platforms?
Slide 36 © 2019 Enterprise Management Associates, Inc.
31%
30%
27%
27%
27%
27%
27%
26%
19%
17%
15%
7%
0% 20% 40%
Network cost
Lack of enterprise capabilities
Unexpected cost
Getting data out of the cloud
Integrating data from on-premises
The cost of moving data
Integration of data stored in different cloud providers
The cost of storing data
The cost of compute resources for integration
Migration to the cloud is too difficult
Getting data into the cloud
We don't do cloud-based data integration

37. What are the biggest challenges you face with
integration for data stored in the cloud?
Slide 37 © 2019 Enterprise Management Associates, Inc.
34%
32%
31%
30%
27%
26%
23%
20%
20%
16%
14%
9%
0% 20% 40%
The cost of compute resources for integration
The cost of storing data
Network cost
Unexpected cost
Lack of enterprise capabilities
Integrating data from on-premises
The cost of moving data
Getting data out of the cloud
Integration of data stored in different cloud providers
Migration to the cloud is too difficult
Getting data into the cloud
We don't store data in the cloud

38. What are the biggest challenges you face in
building and deploying data pipelines?
Slide 38 © 2019 Enterprise Management Associates, Inc.
38%
38%
33%
31%
31%
26%
24%
0% 20% 40%
Difficulty in deploying error-free data pipelines
Too much manual coding
Data engineering resources are scarce
Excessive time spent with break and fix
Data pipelines are too complex
Too many data pipelines
Scaling data pipelines is too difficult

39. Modernization Best Practices:
Automation
Slide 39 © 2019 Enterprise Management Associates, Inc.

40. Which of the following modern capabilities are most
important to your data integration program?
Slide 40 © 2014 Enterprise Management Associates, Inc.
61%
34%
32%
30%
29%
23%
22%
20%
16%
12%
12%
7%
5%
0% 50% 100%
Cloud data management
Scalability in all dimensions
Automation
Data and APIs combined
Simplified management of complex data pipelines
Largescale data loading
Machine learning
Streaming
Data lineage
Replication
Interoperability
Change data capture
Elasticity

41. What aspects of DataOps are most important
to your organization?
Slide 41 © 2019 Enterprise Management Associates, Inc.
48%
38%
35%
30%
27%
26%
25%
23%
23%
21%
13%
7%
6%
0% 10% 20% 30% 40% 50% 60%
Quality control
Automation
Speed of analytics delivery
Accuracy of analytics delivery
Unified data security and privacy
Code and change management
Auditing and monitoring
Speed of iteration
Testing and deployment
Business alignment
Unified data access
Unified orchestration
Democratization of data access

42. What data-related automation is most
important to your organization?
Slide 42 © 2019 Enterprise Management Associates, Inc.
55%
54%
26%
21%
21%
20%
15%
14%
12%
7%
0% 10% 20% 30% 40% 50% 60%
Structuring
Transformation
Governance enforcement
Profiling
Governance detection
Pipeline repairing
Parsing
Cleansing
Lineage
Tagging

43. Modernization Best Practices:
Data Pipeline Automation
Slide 43 © 2019 Enterprise Management Associates, Inc.

44. Which data pipeline management automations
are most important to your organization?
Slide 44 © 2019 Enterprise Management Associates, Inc.
47%
41%
38%
30%
28%
24%
16%
13%
0% 50%
Merge and synchronization
Pipeline bandwidth and processing prioritization
Data pipeline notification and repair
Incremental file system updates
Change data capture
Pipeline conflict resolution
Secondary update pipeline generation
Lineage reconciliation

45. What data pipeline management automation
are you currently using?
Slide 45 © 2019 Enterprise Management Associates, Inc.
43%
38%
38%
32%
32%
30%
26%
20%
0% 20% 40% 60%
Merge and synchronization
Pipeline bandwidth and processing prioritization
Data pipeline notification and repair
Incremental file system updates
Change data capture
Pipeline conflict resolution
Lineage reconciliation
Secondary update pipeline generation

46. What data pipeline management automation
do you plan to use in the next 12 months?
Slide 46 © 2019 Enterprise Management Associates, Inc.
46%
38%
37%
37%
36%
32%
32%
18%
0% 20% 40% 60%
Pipeline bandwidth and processing prioritization (2)
Pipeline conflict resolution (6)
Incremental file system updates (4)
Data pipeline notification and repair
Merge and synchronization
Change data capture
Lineage reconciliation
Secondary update pipeline generation

47. Which of the following best describes how you currently
automate pipeline development, deployment, and
management?
Slide 47 © 2019 Enterprise Management Associates, Inc.
23%
17%
16%
14%
13%
9%
7%
0% 5% 10% 15% 20% 25%
We automate some data pipelines
individually
We automate some data pipelines via
a unified automation tool
We automate most data pipelines via
a unified automation tool
We automate most data pipelines
individually
We automate all data pipelines
individually
We automate all data pipelines via
unified automation tool
We don't automate any data pipelines

48. A Summary of Data Pipeline Automation
Slide 48 © 2019 Enterprise Management Associates, Inc.
57%
26%
17%
0% 20% 40% 60%
We don’t automate pipeline
development and deployment with a
unified automation tool
We automate most or all data
pipelines via a unified automation
tool
We automate some data pipelines
via a unified automation tool

49. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Get the Report at http://bit.ly/3czsdbv
Slide 49 © 2019 Enterprise Management Associates, Inc.