SlideShare a Scribd company logo
1 of 27
Download to read offline
Orchestration and
Operationalization:
The Next Step in your
DataOps Journey
BASIL FARUQUI | DIRECTOR, SOLUTIONS MARKETING | BMC SOFTWARE
BMC Connected Digital Ops
We operationalize innovation, connecting and amplifying hybrid IT with the most innovative portfolio of
infrastructure software, partners, and expertise toenable the Autonomous Digital Enterprise.
ServiceOps
AIOps DevOps DataOps AutonomousOps
We connect across your
entire technology landscape.
We turn systems of record
into systems of action.
We make “Ops” the catalyst
for competitive advantage.
THE VALUE
WE DELIVER
PORTFOLIO
OUTCOMES
SOLUTIONS
Agenda
✓ What’s happening in the world of Data
✓ DataOps and the role of Orchestration
✓ Control-M Platform Capabilities for DataOps
What’s Happening in the
Data World?
THE 2023 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE
“
Big data and analytics software and
cloud services market reached $90
billion in 2021, and it is expected to
more than double by 2026.
Companies Are Investing in Data
IDC: Worldwide Big Data and Analytics Software Forecast, 2022–2026
“
Through 2022, only 15% of use cases leveraging AI techniques (such as ML and
DNNs) and involving edge and IoT environments will be successful.
Future of Intelligence Survey, IDC, August 2021
80 percent of companies’ time in analytics projects is spent on repetitive tasks
such as preparing data, whereas the actual value-added work is limited.
Moreover, just 10 percent of companies believe they have this issue under
control.
McKinsey Survey, 2020
“
And Are Not Seeing Results
DataOps to
the Rescue
Gartner Market
Guide for DataOps
Tools
Source: Gartner® Market Guide for DataOps Tools, December 5, 2022
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S.
and internationally and is used herein with permission. All rights reserved. Gartner does not endorse
any vendor, product or service depicted in its research publications and does not advise technology
users to select only those vendors with the highest ratings or other designation. Gartner research
publications consist of the opinions of Gartner’s Research & Advisory organization and should not be
construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to
this research, including any warranties of merchantability or fitness for a particular purpose.
Enhance Data
Engineering
Practices
Source: 5 Ways to Enhance Your Data Engineering Practices, January 24, 2023
“
… today we are bringing data pipelines from 500 plus different systems and
there are so many platforms and so many events that are coming, data gets
delayed, the model is failing in production, or there’s an outage going on with
a platform, users are not able to consume data, … and when the business
wants to reach out to our teams, the CDO organization, to understand why,
the business gets ping-ponged, between multiple teams. The platform teams
say it’s the DataOps team (there are multiple DataOps teams), or sometimes
they go to the ModelOps teams as well…
Large NA Telco
What Is the Orchestration
Challenge?
Data-Driven Business:
The production at scale challenge
DELIVER
INSIGHTS
INGEST DATA FROM
MULTIPLE SOURCES
STORE
DATA
PROCESS
DATA
Databases Files
ERP/CRM/ Enterprise
Applications
Devices/Sensors Clickstream
Social Media
Azure
Databricks
Azure Data
Factory
Blob Storage
Azure Synapse Analytics
Amazon EMR
AWS Glue
Azure Synapse
© Copyright 2022 BMC Software, Inc.
How Data Engineers See Data Pipelines
Fraud Detection
Predictive Maintenance
Procure to Pay Financial Close
Financial
Reporting Order to Cash
Recommendation
Engines
Predictive
Maintenance
Fraud
Detection
Customer Churn
Predictions
Risk
Modeling
Claims Processing Billing
Core
Banking
Claims
System Brokerage
DELIVER INSIGHTS
INGEST DATA FROM MULTIPLE
SOURCES
STORE DATA
PROCESS
DATA
Interoperability is needed across enterprise and data pipeline orchestrators
Requirements for Production at Scale!
Support
heterogeneous
workflows
SLA management
for workflows
Error handling and
notification
Self-healing and
remediation
End-to-end
visibility
Appropriate UX for
multiple personas
Standards for
running in
production
Support DevOps
practices
Control-M
Platform Capabilities
Machine Learning
Amazon
SageMaker
Azure Machine
Learning
New integrations
every month!!!
App. Workflows
Azure
LogicApps
Airflow
AWS Step
Functions
Google Cloud
Composer
ERP
Oracle
PeopleSoft
Oracle
E-Business Suite
SAP R/3 SAP S/4
HANA
File Transfer
Amazon
S3
SFTP
FTP / FTPS
S3 Comp.
Storage
Azure Blob
Storage
Azure Data Lake
Storage Gen2
GCP
Storage
AS2
Databases
IBM
DB2
Microsoft
SQL Server
Oracle
Database
Sybase Any JDBC
compliant DB
PostgreSQL
Data Processing & Analytics
Amazon
EMR
Azure
HDInsight
Azure
Synapse
Azure
Databricks
Google
Dataproc
Google
Dataflow
Google
BigQuery
Databricks
Snowflake Hadoop
& Spark
Data Integration
Informatica
Cloud Services
AWS Glue
DataBrew
AWS
Glue
Alteryx
Trifacta
Azure
Data Factory
Boomi
Atomsphere
AWS Data
Pipeline
Informatica
PowerCenter
Talend Data
Management
SAP Business
Warehouse
IBM
DataStage
Microsoft
SSIS
dbt
BI & Analytics
Microsoft
Power BI
Qlik
Cloud
Amazon
QuickSight
IBM
Cognos
Miscellaneous
Micro
Focus
SAP Data
Archiving
Veritas
NetBackup
VMware
Communication
Suite
Azure
Backup
Cloud Computing
AWS
Lambda
AWS
Batch
Amazon
EC2
Azure
Functions
Azure
Batch
Azure
VM
Google
Batch
Google
VM
Amazon
ECS
RPA
Automation
Anywhere
UiPath
Control-M Integration Plug-ins
“How late are we going to be?”
“How much time do we have?”
Customer Care
Call Analysis Tracking
SLA Management and Impact Analysis
Access and interact with
Control-M from Python
Control-M
Python Client
Control-M
Workbench
https://hub.docker.com/r/controlm/workbench
Business Users:
Always up to date
Track progress
Order ad-hoc
workflows
“Self-Service has opened a whole new
world for me. The dashboard view is used
every day. I regularly check the status of
our customer billing from home in the
evening on my iPhone or iPad. It provides
me with reassurance that we are hitting
our SLAs.“
—Gillian Glass, Finance Manager, SKY TV
0
File
Transfer
BI
Transformation
Business
Owners
Developers
Data/File
Ops
Application
Ops
Cloud
Ops
Detailed Job monitoring
Detailed FTP monitoring
Predictive SLA monitoring
Workflow Insights
Operational
Views
End-to-End Flow
© Copyright 2022 BMC Software, Inc.
The Data Challenge at Railinc
Operational Excellence
Fleet management
simulation, system
anomaly detection for
improved SLAs and
business decision-making
Data Acquisition
and Management
Real-time Data Ingest
ELT Data Ingest
Metadata Management &
Business Glossary
Intelligent Data
Platform
Enterprise Data Warehouse
Hadoop
Machine Learning
and Analytics
Model Engineering &
Management
Streaming Analytics
Historical Data Analytics
Increased Industry Value
Preventative maintenance,
predictive ERA, improved
models for bad actor and
equipment failure analysis
Business
Outcomes
Daily Data Movement Stats
8M External & internal
web service calls 4.5M External EID
messages 360M Processed
messages
Control-M lets us monitor jobs without having people in front
of a console 24x7. If the solution detects potential delays or
failures, it alerts us immediately. Our customers rely on us to
meet those SLAs. Delayed reports on the Chicago Terminal,
for example, could affect rail operations throughout North
America.
Control-M gives us an intelligent, proactive approach to
keeping processes — and trains — running.
—Robert Redd, Release Engineer, Railinc
Railinc leverages big data and
automation to help keep 1.6M
railcars rolling across 140k miles of
track
Technical Content Resources
Blog: How to orchestrate a data
pipeline on Google Cloud with
Control-M
26
Blog: Orchestration and
Operationalization: The Next Step
In Your Data Journey
Blog: How to orchestrate a data
pipeline on Google Cloud with
Control-M
Control-M Demo: Orchestrating
Real Time Delivery with Artificial
Intelligence
https://bit.ly/3mGQ6HT
https://bit.ly/3HBs1so
https://go.aws/3UySM73
https://bit.ly/3zZFxTu
BMC, the BMC logo, and BMC’s other product names are the exclusive properties of BMC Software, Inc. or its affiliates, are registered or pending registration with the U.S. Patent and
Trademark Office, and may be registered or pending registration in other countries. All other trademarks or registered trademarks are the property of their respective owners. © Copyright
2023 BMC Software, Inc.
About BMC
BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-leading
automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to become an
Autonomous Digital Enterprise that conquers the opportunities ahead.
BMC—Run and Reinvent
www.bmc.com
THANK YOU

More Related Content

Similar to DataOps: Control-M's role in data pipeline orchestration

Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTXCustomer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
tsigitnist02
 

Similar to DataOps: Control-M's role in data pipeline orchestration (20)

Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTXCustomer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
 
How T-Mobile Tamed Metron
How T-Mobile Tamed MetronHow T-Mobile Tamed Metron
How T-Mobile Tamed Metron
 
Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0
 
A Microsoft Approach to Cloud Computing
A Microsoft Approach to Cloud ComputingA Microsoft Approach to Cloud Computing
A Microsoft Approach to Cloud Computing
 
RoMT - Part 2 Marketing Technology Webinar
RoMT - Part 2 Marketing Technology WebinarRoMT - Part 2 Marketing Technology Webinar
RoMT - Part 2 Marketing Technology Webinar
 
CDS + Power Apps
CDS + Power Apps CDS + Power Apps
CDS + Power Apps
 
DataOps - Big Data and AI World London - March 2020 - Harvinder Atwal
DataOps - Big Data and AI World London - March 2020 - Harvinder AtwalDataOps - Big Data and AI World London - March 2020 - Harvinder Atwal
DataOps - Big Data and AI World London - March 2020 - Harvinder Atwal
 
Tech Leaders of DFW presentation by Mirza Chughtai, April 2018
Tech Leaders of DFW presentation by Mirza Chughtai, April 2018Tech Leaders of DFW presentation by Mirza Chughtai, April 2018
Tech Leaders of DFW presentation by Mirza Chughtai, April 2018
 
Greetings david cutler inform and connect
Greetings   david cutler inform and connectGreetings   david cutler inform and connect
Greetings david cutler inform and connect
 
MongoDB World 2019: Managing a Heterogeneous Data Stack with Informatica and ...
MongoDB World 2019: Managing a Heterogeneous Data Stack with Informatica and ...MongoDB World 2019: Managing a Heterogeneous Data Stack with Informatica and ...
MongoDB World 2019: Managing a Heterogeneous Data Stack with Informatica and ...
 
Future of Data Strategy (ASEAN)
Future of Data Strategy (ASEAN)Future of Data Strategy (ASEAN)
Future of Data Strategy (ASEAN)
 
IBM Cloud pak for data brochure
IBM Cloud pak for data   brochureIBM Cloud pak for data   brochure
IBM Cloud pak for data brochure
 
Tableau reseller partner in Djibouti Bilytica Best business Intelligence Comp...
Tableau reseller partner in Djibouti Bilytica Best business Intelligence Comp...Tableau reseller partner in Djibouti Bilytica Best business Intelligence Comp...
Tableau reseller partner in Djibouti Bilytica Best business Intelligence Comp...
 
Tableau reseller partner in Fiji Bilytica Best business Intelligence Company ...
Tableau reseller partner in Fiji Bilytica Best business Intelligence Company ...Tableau reseller partner in Fiji Bilytica Best business Intelligence Company ...
Tableau reseller partner in Fiji Bilytica Best business Intelligence Company ...
 
Tableau reseller partner in Brunei Bilytica Best business Intelligence Compan...
Tableau reseller partner in Brunei Bilytica Best business Intelligence Compan...Tableau reseller partner in Brunei Bilytica Best business Intelligence Compan...
Tableau reseller partner in Brunei Bilytica Best business Intelligence Compan...
 
Tableau reseller partner in Cambodia Bilytica Best business Intelligence Comp...
Tableau reseller partner in Cambodia Bilytica Best business Intelligence Comp...Tableau reseller partner in Cambodia Bilytica Best business Intelligence Comp...
Tableau reseller partner in Cambodia Bilytica Best business Intelligence Comp...
 
Tableau reseller partner in Bulgaria Bilytica Best business Intelligence Com...
 Tableau reseller partner in Bulgaria Bilytica Best business Intelligence Com... Tableau reseller partner in Bulgaria Bilytica Best business Intelligence Com...
Tableau reseller partner in Bulgaria Bilytica Best business Intelligence Com...
 
Tableau reseller partner in Egypt Bilytica Best business Intelligence Company...
Tableau reseller partner in Egypt Bilytica Best business Intelligence Company...Tableau reseller partner in Egypt Bilytica Best business Intelligence Company...
Tableau reseller partner in Egypt Bilytica Best business Intelligence Company...
 
Tableau reseller partner in Estonia Bilytica Best business Intelligence Compa...
Tableau reseller partner in Estonia Bilytica Best business Intelligence Compa...Tableau reseller partner in Estonia Bilytica Best business Intelligence Compa...
Tableau reseller partner in Estonia Bilytica Best business Intelligence Compa...
 
Tableau reseller partner in Croatia Bilytica Best business Intelligence Compa...
Tableau reseller partner in Croatia Bilytica Best business Intelligence Compa...Tableau reseller partner in Croatia Bilytica Best business Intelligence Compa...
Tableau reseller partner in Croatia Bilytica Best business Intelligence Compa...
 

Recently uploaded

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 

Recently uploaded (20)

WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
%in Soweto+277-882-255-28 abortion pills for sale in soweto
%in Soweto+277-882-255-28 abortion pills for sale in soweto%in Soweto+277-882-255-28 abortion pills for sale in soweto
%in Soweto+277-882-255-28 abortion pills for sale in soweto
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
 
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public AdministrationWSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
 
WSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security ProgramWSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security Program
 
AI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplateAI & Machine Learning Presentation Template
AI & Machine Learning Presentation Template
 
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
 
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
 
WSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration ToolingWSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration Tooling
 
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of TransformationWSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the past
 
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
 
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
 
WSO2Con2024 - Software Delivery in Hybrid Environments
WSO2Con2024 - Software Delivery in Hybrid EnvironmentsWSO2Con2024 - Software Delivery in Hybrid Environments
WSO2Con2024 - Software Delivery in Hybrid Environments
 

DataOps: Control-M's role in data pipeline orchestration

  • 1. Orchestration and Operationalization: The Next Step in your DataOps Journey BASIL FARUQUI | DIRECTOR, SOLUTIONS MARKETING | BMC SOFTWARE
  • 2. BMC Connected Digital Ops We operationalize innovation, connecting and amplifying hybrid IT with the most innovative portfolio of infrastructure software, partners, and expertise toenable the Autonomous Digital Enterprise. ServiceOps AIOps DevOps DataOps AutonomousOps We connect across your entire technology landscape. We turn systems of record into systems of action. We make “Ops” the catalyst for competitive advantage. THE VALUE WE DELIVER PORTFOLIO OUTCOMES SOLUTIONS
  • 3. Agenda ✓ What’s happening in the world of Data ✓ DataOps and the role of Orchestration ✓ Control-M Platform Capabilities for DataOps
  • 4. What’s Happening in the Data World?
  • 5. THE 2023 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE
  • 6. “ Big data and analytics software and cloud services market reached $90 billion in 2021, and it is expected to more than double by 2026. Companies Are Investing in Data IDC: Worldwide Big Data and Analytics Software Forecast, 2022–2026
  • 7. “ Through 2022, only 15% of use cases leveraging AI techniques (such as ML and DNNs) and involving edge and IoT environments will be successful. Future of Intelligence Survey, IDC, August 2021 80 percent of companies’ time in analytics projects is spent on repetitive tasks such as preparing data, whereas the actual value-added work is limited. Moreover, just 10 percent of companies believe they have this issue under control. McKinsey Survey, 2020 “ And Are Not Seeing Results
  • 9. Gartner Market Guide for DataOps Tools Source: Gartner® Market Guide for DataOps Tools, December 5, 2022 GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s Research & Advisory organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
  • 10. Enhance Data Engineering Practices Source: 5 Ways to Enhance Your Data Engineering Practices, January 24, 2023
  • 11. “ … today we are bringing data pipelines from 500 plus different systems and there are so many platforms and so many events that are coming, data gets delayed, the model is failing in production, or there’s an outage going on with a platform, users are not able to consume data, … and when the business wants to reach out to our teams, the CDO organization, to understand why, the business gets ping-ponged, between multiple teams. The platform teams say it’s the DataOps team (there are multiple DataOps teams), or sometimes they go to the ModelOps teams as well… Large NA Telco
  • 12. What Is the Orchestration Challenge?
  • 13. Data-Driven Business: The production at scale challenge DELIVER INSIGHTS INGEST DATA FROM MULTIPLE SOURCES STORE DATA PROCESS DATA Databases Files ERP/CRM/ Enterprise Applications Devices/Sensors Clickstream Social Media Azure Databricks Azure Data Factory Blob Storage Azure Synapse Analytics Amazon EMR AWS Glue Azure Synapse © Copyright 2022 BMC Software, Inc.
  • 14. How Data Engineers See Data Pipelines Fraud Detection Predictive Maintenance
  • 15. Procure to Pay Financial Close Financial Reporting Order to Cash Recommendation Engines Predictive Maintenance Fraud Detection Customer Churn Predictions Risk Modeling Claims Processing Billing Core Banking Claims System Brokerage DELIVER INSIGHTS INGEST DATA FROM MULTIPLE SOURCES STORE DATA PROCESS DATA Interoperability is needed across enterprise and data pipeline orchestrators
  • 16. Requirements for Production at Scale! Support heterogeneous workflows SLA management for workflows Error handling and notification Self-healing and remediation End-to-end visibility Appropriate UX for multiple personas Standards for running in production Support DevOps practices
  • 18. Machine Learning Amazon SageMaker Azure Machine Learning New integrations every month!!! App. Workflows Azure LogicApps Airflow AWS Step Functions Google Cloud Composer ERP Oracle PeopleSoft Oracle E-Business Suite SAP R/3 SAP S/4 HANA File Transfer Amazon S3 SFTP FTP / FTPS S3 Comp. Storage Azure Blob Storage Azure Data Lake Storage Gen2 GCP Storage AS2 Databases IBM DB2 Microsoft SQL Server Oracle Database Sybase Any JDBC compliant DB PostgreSQL Data Processing & Analytics Amazon EMR Azure HDInsight Azure Synapse Azure Databricks Google Dataproc Google Dataflow Google BigQuery Databricks Snowflake Hadoop & Spark Data Integration Informatica Cloud Services AWS Glue DataBrew AWS Glue Alteryx Trifacta Azure Data Factory Boomi Atomsphere AWS Data Pipeline Informatica PowerCenter Talend Data Management SAP Business Warehouse IBM DataStage Microsoft SSIS dbt BI & Analytics Microsoft Power BI Qlik Cloud Amazon QuickSight IBM Cognos Miscellaneous Micro Focus SAP Data Archiving Veritas NetBackup VMware Communication Suite Azure Backup Cloud Computing AWS Lambda AWS Batch Amazon EC2 Azure Functions Azure Batch Azure VM Google Batch Google VM Amazon ECS RPA Automation Anywhere UiPath Control-M Integration Plug-ins
  • 19. “How late are we going to be?” “How much time do we have?” Customer Care Call Analysis Tracking SLA Management and Impact Analysis
  • 20. Access and interact with Control-M from Python Control-M Python Client
  • 22. Business Users: Always up to date Track progress Order ad-hoc workflows “Self-Service has opened a whole new world for me. The dashboard view is used every day. I regularly check the status of our customer billing from home in the evening on my iPhone or iPad. It provides me with reassurance that we are hitting our SLAs.“ —Gillian Glass, Finance Manager, SKY TV 0
  • 23. File Transfer BI Transformation Business Owners Developers Data/File Ops Application Ops Cloud Ops Detailed Job monitoring Detailed FTP monitoring Predictive SLA monitoring Workflow Insights Operational Views End-to-End Flow © Copyright 2022 BMC Software, Inc.
  • 24. The Data Challenge at Railinc Operational Excellence Fleet management simulation, system anomaly detection for improved SLAs and business decision-making Data Acquisition and Management Real-time Data Ingest ELT Data Ingest Metadata Management & Business Glossary Intelligent Data Platform Enterprise Data Warehouse Hadoop Machine Learning and Analytics Model Engineering & Management Streaming Analytics Historical Data Analytics Increased Industry Value Preventative maintenance, predictive ERA, improved models for bad actor and equipment failure analysis Business Outcomes Daily Data Movement Stats 8M External & internal web service calls 4.5M External EID messages 360M Processed messages
  • 25. Control-M lets us monitor jobs without having people in front of a console 24x7. If the solution detects potential delays or failures, it alerts us immediately. Our customers rely on us to meet those SLAs. Delayed reports on the Chicago Terminal, for example, could affect rail operations throughout North America. Control-M gives us an intelligent, proactive approach to keeping processes — and trains — running. —Robert Redd, Release Engineer, Railinc Railinc leverages big data and automation to help keep 1.6M railcars rolling across 140k miles of track
  • 26. Technical Content Resources Blog: How to orchestrate a data pipeline on Google Cloud with Control-M 26 Blog: Orchestration and Operationalization: The Next Step In Your Data Journey Blog: How to orchestrate a data pipeline on Google Cloud with Control-M Control-M Demo: Orchestrating Real Time Delivery with Artificial Intelligence https://bit.ly/3mGQ6HT https://bit.ly/3HBs1so https://go.aws/3UySM73 https://bit.ly/3zZFxTu
  • 27. BMC, the BMC logo, and BMC’s other product names are the exclusive properties of BMC Software, Inc. or its affiliates, are registered or pending registration with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other trademarks or registered trademarks are the property of their respective owners. © Copyright 2023 BMC Software, Inc. About BMC BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-leading automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to become an Autonomous Digital Enterprise that conquers the opportunities ahead. BMC—Run and Reinvent www.bmc.com THANK YOU