1. Migration from an on-prem HDFS system to
Microsoft Azure Databricks

2. What is Databricks?
● Unified Data Platform: Databricks provides a unified platform for data engineering, data science, and business
analytics. It integrates with various data sources and tools, making it easier to collaborate and work on different
aspects of a data project.
● Scalability and Performance: Databricks leverages Apache Spark, a powerful open-source distributed processing
framework, to handle large-scale data processing. It can scale horizontally to handle massive data volumes and perform
computations in a distributed and parallel manner, enabling faster processing and analysis.
● Data Engineering Capabilities: Databricks simplifies data engineering tasks such as data ingestion, transformation, and
cleaning. It offers a user-friendly interface and a wide range of built-in connectors and tools to facilitate these tasks,
reducing the time and effort required for data preparation.
● Data Science and Machine Learning: Databricks provides a collaborative environment for data scientists and machine
learning engineers to build, train, and deploy models. It supports popular languages like Python and R and integrates
with libraries and frameworks such as TensorFlow, PyTorch, and scikit-learn, making it easier to develop and experiment
with machine learning algorithms.
● Advanced Analytics and Visualization: Databricks offers built-in tools and libraries for advanced analytics and
visualization, enabling users to explore and analyze data effectively. It supports SQL queries, interactive notebooks, and
visualizations, making it accessible to both technical and non-technical users.
● Cloud-Native and Managed Service: Databricks is a cloud-native platform and is available as a managed service on
major cloud providers such as AWS, Azure, and Google Cloud. It provides easy setup, scalability, and automatic
infrastructure management, reducing the operational overhead for organizations.
● Collaboration and Integration: Databricks allows teams to collaborate effectively on data projects. Multiple users can
work on the same notebooks, share insights, and track changes. It also integrates with other popular tools and services,
such as version control systems, business intelligence tools, and data warehouses, facilitating seamless workflows.

3. Why Databricks
● Apache Spark Integration: Databricks is tightly integrated with Apache Spark, an extremely powerful and versatile open-source
distributed computing framework. Databricks provides an optimized and managed version of Spark, allowing users to leverage
its capabilities without worrying about infrastructure setup, configuration, and maintenance.
● Auto-scaling and Cost Optimization: Databricks offers auto-scaling capabilities that automatically provision and deprovision
compute resources based on workload demand. This ensures optimal resource utilization and cost efficiency, as you only pay for
the resources you actually need. Databricks also provides cost optimization features such as instance scaling and job scheduling
to help manage and reduce cloud costs.
● Collaboration and Sharing: Databricks provides collaborative features that enable teams to work together effectively. Users can
share notebooks, libraries, and insights with colleagues, facilitating knowledge sharing and streamlined collaboration. It also
supports version control integration, allowing teams to manage changes and collaborate on code development.
● MLflow Integration: MLflow is an open-source platform for managing the machine learning lifecycle. Databricks offers native
integration with MLflow, providing a centralized repository for tracking experiments, managing models, and deploying them into
production. This integration simplifies the process of building, versioning, and deploying machine learning models.
● Delta Lake: Databricks includes Delta Lake, an open-source storage layer that adds reliability, ACID transactions, and versioning
capabilities to data lakes. Delta Lake provides features like schema enforcement, data lineage, and time travel queries, making
data management and governance easier. It ensures data consistency and reliability in large-scale data lake environments.
● Streamlined Deployment and Integration: Databricks provides seamless integration with various data sources, data warehouses,
and other cloud services. It supports connectors for popular data storage systems, such as AWS S3, Azure Data Lake Storage,
and Google Cloud Storage. Databricks also integrates with popular data integration tools like Apache Kafka and AWS Glue,
simplifying the process of ingesting and processing data from different sources.
● Security and Compliance: Databricks prioritizes security and compliance with industry standards. It offers features like granular
access controls, encryption at rest and in transit, and support for compliance frameworks like SOC 2, HIPAA, and GDPR.
Databricks also provides auditing and monitoring capabilities to help organizations track and analyze user activities and ensure
data protection.

4. Summary: This migration plan is based on a real-world client migrating from an on-prem
environment to a cloud provider
Current Situation: The immediate needs are to migrate their core applications to the cloud. Their
existing environment uses Kafka + Spark and is on-prem. Microsoft Azure as selected as the cloud
provider and Confluent for real-time data processing. Databricks is the analytics platform.
Proposed Plan: The migration plan addresses the basic requirements and the use cases needed to
migrate from an on-prem environment to a cloud network - Microsoft Azure and Databricks is the
analytics platform
Scenario

5. Use Cases

6. 1. Real-time data processing:
This plan includes moving core on-prem applications from an on-prem HDFS system to Azure Databricks and enabling
real-time analytics for analytics and reporting metrics on Confluent on the Microsoft Azure Databricks Platform
1. Batch processing:
It also covers the migration of batch processing workloads from HDFS to Databricks as a unified platform levering
Apache Spark to process large-scale data to help streamline and optimize batch processing workloads
1. Analytics and Machine Learning:
The plan includes moving the analytics and machine learning workloads from on-prem HDFS to Databricks, as a
collaborative and integrated workspace for data engineering, machine learning, and analytics enabling advanced
analytics and machine learning on the data, and gain deeper insights and value from it
1. Cloud migration plan:
This plan recommends a phased approach to migration that follows an Agile test and validation methodology for an
end-to-end transformation of data sources from source environment to the target environment
1. Data management and Governance:
The plan recommends data management and governance processes to be followed for a Cloud environment fully
supported by an Azure Databricks environment
Use Cases covered in the proposed plan

7. High-level Migration Architecture

8. Databricks(Kafka on Spark)
(Spark+Kafka)
High Level Migration Architecture
Source
Apps
Destinati
on Apps
HDFS ADLS
Azure Migration Services
Kafka Connector
(Confluent/Databricks)
Azure Data
Factory
Azure Site Recovery
(Disaster
Management)
Confluent
QoS
*This architecture plan supports a lift and shift approach to migration
Databricks
Analytics
and MLOps
QoS
Metrics

9. Design Considerations

10. Key Elements considered for Migration
Requirement Proposed Solution Description
Cloud Provider Microsoft Azure Microsoft Azure is the recommended cloud provider, providing a reliable and
scalable platform that supports the customer's data analytics and reporting
requirements.
Data Ingestion and
Processing
Kafka+Confluent Kafka integrated with the Confluent platform would be used for real-time data
processing and ingestion, enabling efficient data processing and integration
with the customer's existing on-premises systems.
Data Storage Azure Data Lake Storage Azure Data Lake Storage would be used for data storage, providing a highly
available and scalable storage solution that meets the customer's data
retention and retrieval requirements.
Analytics and Machine
Learning
Databricks The Databricks analytics platform would be used for data analytics and
machine learning, enabling the customer to gain insights and make informed
decisions based on their data.
Network and Security Azure Virtual Networks Azure Virtual Networks would be used for secure communication between the
various components of the architecture, ensuring that data is protected at all
times. Access controls and encryption would also be implemented to ensure
that data is secure and compliant with relevant regulations and policies.
Monitoring and Alerting Confluent + Databricks Confluent integrated with Databricks Analytics can be used for monitoring and
alerting in the Azure Databricks migration process.

11. Primary Azure Databricks Cloud Components
Azure Cloud Component Description
Azure Data Factory This is a fully managed, cloud-based data integration service that enables you to move and transform data from various
sources to various destinations, including Databricks. It provides a comprehensive set of tools and services for building
data integration pipelines, including data movement, data transformation, and data orchestration. You can use Azure
Data Factory to migrate data from HDFS on-prem to Databricks, and also to integrate data from Confluent into the
Databricks environment.
Azure Databricks This is a fast, easy, and collaborative Apache Spark-based analytics platform that provides a fully managed, cloud-
based environment for data engineering, data science, and machine learning. Azure Databricks can be used to process
and analyze data from HDFS on-prem, and also to process and analyze real-time data from Confluent.
Azure Virtual Network Azure Virtual Network is a logical isolation of the Azure cloud dedicated to an organization's network. It provides secure
communication between Azure resources, on-premises resources, and the internet. Virtual Network is recommended
for a Databricks migration to ensure secure communication between the Databricks workspace and other resources.
Databricks Kafka/Azure
event hubs
Azure Events Hub is a fully managed, real-time data ingestion service that enables you to collect, process, and analyze
data from various sources, including Confluent. Azure Event Hubs can be used to integrate real-time data from
Confluent into the Databricks environment, and also to stream data from Databricks to downstream systems for real-
time reporting.
Databricks Kafka can be used as a streaming option to do additional analytics of data on the Databricks platform
Azure ADLS Gen2 Azure Data Lake Storage Gen2 is a highly scalable and cost-effective data lake solution that can be used to store and
analyze structured and unstructured data. It provides native integration with Databricks and can be used for data
processing, machine learning, and analytics use cases.

12. Phased Approach to Migration
Discovery &
Assessment
The first step in our
plan is to conduct a
discovery and
assessment of the
existing environment,
including data sources,
applications, and
infrastructure. This will
help us identify any
potential challenges
and risks that may
impact the migration.
Design &
Architecture
Based on the findings
from the discovery and
assessment, we will
design an Azure
Databricks architecture
that meets their data
requirements. Our
proposed architecture
includes using Azure
Virtual Networks for
secure communication,
Azure Data Lake Storage
for data storage
supporting Confluent for
real-time data processing
which is their tool of
choice. We will leverage
Databricks platform for
data analytics and
Implementation
We will begin
implementing the
solution in a phased
approach, starting with a
development
environment, followed by
a testing environment,
and then a production
environment. We will
work closely with the SI
Partners to ensure a
smooth migration
process, and provide
support to the delivery
team during this process.
Deployment &
Production
After successful
testing and validation,
we will deploy the
solution to production.
We will work with the
customer's delivery
team to ensure a
smooth transition to
the new environment,
and provide training
and support to ensure
that they are equipped
to manage the new
environment.
Testing &
Validation
Once the solution is
implemented, we will
conduct extensive
testing and validation
to ensure that it meets
all the data quality
requirements. We will
perform functional
testing, integration
testing, and
performance testing to
ensure that the
solution is operating
as expected.

13. Migration Roadmap
Q3 2024
Q4 Q1
Q2
On-Prem to
Azure
Databricks
Migration
2023 Q2
We
are
here
Initial Plan Sign-
off
Discovery &
Assessment
Design &
Architecture
Design Sign-off &
Implementation Planning
Implementation
Dev
Environment
/Testing
and Validation
Test
Environment
/Testing
and Validation
Testing Validation
Sign-off & Prod
Deployment
Go Live
* This is a high level Roadmap and can be worked in more detail in the Comprehensive
Migration Plan

14. Discovery & Assessment (Month 1-2)
● Conduct a comprehensive discovery and assessment of the existing environment to identify potential
challenges and risks that may impact the migration.
● Review and validate the data requirements and ensure that they are achievable within the proposed
architecture and technology stack.
● Work with the networking teams and SI Partners to identify and prioritize the applications and data sources
to be migrated.
● Identify roles and responsibilities and a communication plan within teams to highlight specific areas of
ownership and expertise
Questions to address (examples):
a. What is the size, nature of the data that needs to be migrated?
b. What is a high-level of data ingress and egress needs to identify a streaming solution(Confluent Kafka vs.
Azure Events Hub vs. Databricks Kafka)?
c. Are code repository setup locations identified? Are there 3P integrations like JIRA Zendesk?
d. Who is responsible for the application inventory gathering and task assignments?

15. Design & Architecture (Month 3-5)
● Based on the findings from the discovery and assessment, design a cloud-based architecture that meets all data
requirements, using Azure Virtual Networks for secure communication, Azure Data Lake Storage for data storage,
and Confluent for real-time data processing.
● Work with Mega Division’s networking team and SI Partners to review and approve the proposed architecture and
ensure that it aligns with their requirements and constraints.
● Determine the specific requirements for the Databricks analytics platform, including data ingestion, transformation,
and analytics.
Questions to address (examples):
a. What are the key components used?
b. Are there any non-standard architectures or procedures used?
c. Are applications enabled with CICD
d. What parts of code will be in notebooks vs what part in Jars/Cron jobs
e. Are there any monitoring tools or logging tools currently that need migration?

16. Implementation (Month 6-9)
● Implement the solution in a phased approach, starting with a development environment, followed by a testing
environment, and then a production environment.
● Work with the SI Partners to deploy the Confluent and Databricks clusters on Azure and configure them to meet the
customer's requirements.
● Migrate the applications and data sources in a structured manner, using a combination of manual and automated
methods, to ensure minimal disruption to the customer's operations.
● Work with the customer's networking team and SI Partners to configure the necessary security and access controls
for the new environment.
Questions to address (examples):
a. Are clusters deployed?
b. Security setup in place?
c. Which notebook folders are open for which users?
d. How do users share code and data?
e. Are there any overlap issues in using the infrastructure?

17. Testing & Validation (Month 7-9)
● Conduct extensive testing and validation of the solution to ensure that it meets the customer's data requirements
and performance expectations.
● Perform functional testing, integration testing, and performance testing to ensure that the solution is operating as
expected and meeting all requirements.
● Work with the customer's networking team and SI Partners to conduct user acceptance testing and ensure that they
are satisfied with the new environment.
Questions to address (examples):
a. How do we test the output accuracy?
b. Do we need to write code to automatically test results on a new platform?
c. What is the testing process? Are test scripts prepared and ready?
d. How are we doing on key metrics in the new Dev environment compared to native
e. How are we doing on key metrics in the new Test environment compared to native
*The key metrics to test and validate are discussed in the later sections
of the proposed plan

18. Production Deployment (Month 10-11)
● After successful testing and validation, deploy the solution to production in a controlled manner, using a staged rollout to
minimize disruption to the current operations.
● Work with the delivery team and SI Partners to ensure a smooth transition to the new environment and provide training and
support to ensure that they are equipped to manage the new environment.
● Monitor the environment closely in the initial days after deployment and address any issues or concerns that arise.
Questions to address (examples):
a. Measure and understand if KPIs are met.
b. If not met, introspect, and identify what needs to be done.
c. Are basic essential KPIs met, so that we can go live and address the technical debt?
d. Identify all technical debt, document
e. Define a plan to address technical debt.
f. Is a new system up and running for sufficient time to hand over for production support.

19. Go Live! (Month 12)
Key Steps for Go Live:
○ This time will be dedicated to all events related to a Go-Live into the new environment:
○ Dedicate a core team to monitor the end-to end transformations in the new system
○ Monitor the Key KPIs highlighted to track the key metrics and measure them against set
benchmarks and standards
○ Identify any possible lags or hiccups to streamline them in future iterations
Things to highlight in this time:
○ Measure and understand if KPIs are met.
○ If not met, introspect, and identify what needs to be done.
○ Are basic essential KPIs met, so that we can go live and address the technical debt?
○ Identify all technical debt, document
○ Define a plan to address technical debt.
○ Is a new system up and running for sufficient time to hand over for production support.

20. Alliances and Partnerships

21. Alliances and Partnerships
1. Partners:
● Engage with the SI Partners involved with the migration team to assist with key system integrations and sources of truth.
● Collaborate closely with the SI Partners to leverage their expertise in the new environment and specific technologies to
expedite the implementation.
● Ensure that the SI Partners are trained on the proposed architecture and technology stack to facilitate a smooth
migration.
● Work with Microsoft and Confluent, and potentially other technology partners, to leverage their expertise and support in
the migration process if needed.
2. Professional Services:
● Engage with Databricks' professional services team to ensure that the analytics platform is configured optimally and that
the customer's data sources are properly integrated.
● Work with the SI Partners and network team to provide additional professional services support, as needed, for specific
aspects of the migration project.
● Leverage our own professional services team to supplement the delivery team, as needed, to accelerate this
implementation and provide additional expertise and support.
2. Training:
● Provide training to the customer's delivery team on the proposed architecture and technology stack to ensure that they
are equipped to manage the new environment.
● Leverage Databricks' training resources to upskill the customer's delivery team on the analytics platform and related
technologies.
● Work with the SI Partners and network teams to identify any additional training needs for their personnel and provide
training or facilitate access to relevant training resources.
2. Power Users:
○ Power users can provide valuable insights into the data structure, schema, and quality of the data, as well as any
challenges or issues that may arise during the deployment and migration process.
○ They can help in data profiling, mapping and transformation, testing and validation, and training and knowledge transfer
to ensure a successful migration.

22. Potential Risks and Compliance
Considerations

23. Security, Compliance and Other Risks and Considerations
Performance & Availability
04
● System availability and performance of the new cloud
environment can be tracked using alert and monitoring
tools provided by Confluence. Azure Monitoring can also
be used to monitor the system performance during the
migration process
Integration Complexity
02
● Inventory all the processes that need to be migrated and
the systems that need to be integrated on the new Cloud
environment in the Discovery & Assessment phase
● Create a compatibility matrix of all the existing systems
on-prem and map them to the new systems in the Azure
Databricks Cloud environment
Data Security & Compliance
01
● Work closely with SI Partners and network team to
establish security based on industry standards
● Implement appropriate access controls, encryption, and
monitoring to ensure that data is protected at all times
Technical Compatibility
03
● Besides creating the complexity matrix mentioned above
SI Partners and the Delivery team needs to work closely to
establish the new platform standards in the Design and
Architecture phase

24. Metrics to Consider

25. Operational Metrics to consider in the migration process
Migration Progress
System
Availability
Performance &
Scalability
Data
Quality
User
Adoption
Cost
Optimiza
tion
1.Migration Progress: This metric would track the overall progress of the migration
project, including the number of applications and data sources migrated and the
percentage of migration completion. This metric would provide with visibility into the
project's status and help them assess whether the project is on track to meet their
timeline.
2.System Availability: This metric would track the availability of the new cloud-based
environment, including the uptime of the systems and applications. This metric would
be important, as it would ensure that the new environment is meeting their QoS
requirements and would provide visibility into any potential downtime or disruptions.
3.Data Quality: This metric would track the quality of the migrated data, including data
accuracy, completeness, and consistency. This metric would be important as it would
ensure that the data is reliable and meets their business needs.
4.Performance and Scalability: These metrics would track the performance and
scalability of the new environment, including response times, throughput, and system
utilization. These metrics would be important to, as they would ensure that the new
environment is meeting their QoS requirements and would provide visibility into any
potential performance or scalability issues.
5.User Adoption: This metric would track the adoption of the new environment by
end-users, including the number of users accessing the system and their level of
engagement. This metric would be important to, as it would ensure that the new
environment is meeting their business needs and would provide visibility into any
potential user adoption issues.
6.Cost Optimization: This metric would track the cost savings achieved by migrating
to the new cloud-based environment, including reductions in infrastructure costs and
operational expenses. This metric would be important to, as it would ensure that the
migration project is delivering the expected return on investment.
Note: These metrics are suggested for the Tracking & Reporting of the
migration process and can leverage the Kafka+Confluent features for
specifics

26. Additional Requirements

27. Additional Requirements and Considerations
The following might help to build a more comprehensive plan to kick-off the migration
Current Performance and
Availability Metrics
Information on the current performance
and availability metrics would help us
understand the QoS requirements and
identify any potential issues or gaps in
the current environment. This would
enable us to develop a plan that
addresses these issues and ensures
that the new environment meets all the
performance and availability
requirements.
Infrastructure &
Architecture Diagrams
Detailed diagrams of the existing
infrastructure and architecture will
help understand the current
environment and identify any
potential compatibility or integration
issues to develop a detailed
migration plan that addresses any
technical challenges.
Detailed Scope of Work
A detailed scope of work would
provide an overview of the business
requirements, including the specific
applications and data sources that
need to be migrated and complexity
of the project and enable us to
develop an appropriate plan.
Budget & Timeline
Constraints
Information on the budget and
timeline constraints would enable
us to develop a plan that meets
requirements while delivering the
expected ROI. This would also help
identify any potential trade-offs that
may need to be made during the
project.
Data & Application
Dependencies
Information on data and application
dependencies would enable us to identify
any interdependencies that may impact
the migration process. This would enable
us to develop a detailed plan that
addresses these dependencies and
ensures a smooth migration process.
Note: It will also be
beneficial if the
core Delivery team
works closely with
Databricks
Professional
Services, SI
Partners and all
other stakeholders
of the Mega
Division to identify
roles and
responsibilities
throughout the
migration process.

28. Thank you!