Modernising any system is a comprehensive task. Every step has to be estimated, appropriately planned, then carefully executed and verified. Data with its dependencies are the common denominator in almost every case and crucial in understanding the whole initiative. In this webinar, experts from Profinit and Manta will present their approach to resolving data-related challenges while modernising software systems using Profinit Modernisation Framework in collaboration with Manta tools.

1. &
Understand your data dependencies
Key enabler to efficient modernisation
May 19th, 2021

2. &
2
Contents
1. Organisation of the webinar
2. Challenge #1 - your data dependencies
3. Profinit Modernisation Framework in Detail
4. MANTA showcase
5. Q&A

3. &
3
mpetrik@profinit.eu
www.linkedin.com/in/mpetrik-41442895
@Profinit_SW
Profinit: Michal Petřík
Preferred employer of extraordinary talents
› Supporting start-ups, student competitions
› Computer science education for kids
20+ years in the business
and growing
Unparalleled collaboration with Top CompSci Universities
› Five software engineering focused modules
› Applied science (research groups and grants)

4. &
MANTA: Jan Ulrych
› MANTA is a unified lineage platform allowing information users to
understand complex information systems to augment and optimize
existing processes, including:
– Data Quality
– DataOps
– Modernizations and Data Privacy
– Data Governance
jan.ulrych@getmanta.com
https://www.linkedin.com/in/janulrych/
https://twitter.com/get_manta

5. &
5
Organisation of the webinar
› 16:00 - 16:10 CEST / 10:00 AM - 10:10 AM EDT - Introduction
› 16:10 - 16:35 CEST / 10:10 AM - 10:35 AM EDT - Profinit
– Why your data dependencies are the key during modernisation
– How Profinit Modernisation Framework using MANTA addresses the challenge
› 16:35 - 17:00 CEST / 10:35 - 11:00 EDT- MANTA
– MANTA introduction
– MANTA show case – data lineage
› 17:00 - 17:15 CEST / 11:00 AM - 11:15 AM EDT - Q&A
– Through Sli.do platform anytime during the presentations

6. &
Questions and polls
https://www.sli.do
event #2854

7. &
Challenge #1
Your data dependencies

8. Regarding modernisation you are
ⓘ Start presenting to display the poll results on this slide.

9. What is/was the driver to start it?
ⓘ Start presenting to display the poll results on this slide.

10. &
10
Why is it about data?

11. &
11
Motivation
› More than 80% of data migration projects run overtime and/or over
budget. Cost overruns average 30%. Time overruns average 41%.”
Bloor Group
› “83% of data migration projects either fail or exceed their budgets
and schedules.”
Gartner

12. How many integrations do your core
systems have?
ⓘ Start presenting to display the poll results on this slide.

13. &
14
Where does it end?
Are we talking about monolithic systems?

14. &
15
› The key is to identify friction points and modernize them
– Not possible without understanding the data model and dependencies
Continuous Systems Modernisation

15. &
Profinit
Modernisation
Framework

16. &
17
Profinit Modernisation Framework
Assessment processes
Defines assessment processes
to analyse the challenge and
choose the right modernisation
approach.
Analytical tools
Uses best-in-class tooling
together with unique in-house
solutions to get a complex
overview of the challenge.
Software engineering excellence
Continuously applies software
engineering and computer science
best practices to achieve stunning
results and add business value.
Iterative, incremental & safe
Applies well-planned small steps to
achieve great goals while enabling
full customer control over the
long-term modernisation process.
Changes with future in mind
Proposes architectural, design,
technological and process
changes with respect to current
trends. Always plans for the
upcoming years, not months.

17. &
18
Assessment process

18. &
19
Sample schedule
› Usually start with a series of workshops (2 times 2 days, etc.)
› Rough definition of the scope and business areas pre-selection
1. Iteration
› The system – scope / domain
› Logical parts
› Assessment
› Documentation
› Architecture/design
› Source codes
› Environment, DevOps
› Team
› Key friction points identification
› Estimation of the next iteration
› Toolbox update
2. Iteration
› Focused analyses
(friction points)
› Usually performed by:
› Team leader
› BE team: 2 persons
› FE team: 2 persons
› PoC/PoT for selected friction
points
› Enhanced estimation
› Toolbox update
3. Iteration
› Sample deliverables
› BE / FE parts
› Data
› The next steps definition
› Friction points
› Enhancements
› Timing and estimation
› Team proposal & required
collaboration capacity
› Results presentation and
assessment handover
3 weeks 3 weeks 3 weeks
Go/NoGo point Go/NoGo point The next steps

19. &
20
Assessment process
› Key attribute here is the size & complexity
of the problem
› Implies the ability to estimate and plan
 reasonable expectations and delays reduction

20. &
21
Analytical tools
› Best in class tooling & custom made utilities
– Source code static analysis
– Data analysis
– Data transformation and visualisation
– Data tracing
– …
› Every project is unique  no „one size fits all“ approach exists
– Custom made tools based on collected metadata
– MANTA is the perfect source for this area

21. &
22
Software engineering excellence
› Automation is the key to success
› Every step shall be repeatable, testable and revertible
– You cannot predict future and be prepared for everything…
› Pre / post modernisation state validation helps a lot
– Migration from one technology / platform to another
– Behaviour before / after friction point elimination
– …
› Finding and preparing proper testing data might be a challenge too
–  data lineage and dependencies analysis helps significantly

22. &
23
Iterative, incremental & safe
› Big bang projects usually end with
big spectacular failures
› Relatively short iterations and increments should be always chosen
– Or at least at the beginning of the project  e.g. „fail-fast“ approach
› Before each iteration/increment do:
– Friction point analysis (validation)
– Impact analysis
– Estimation
– Pre / post validation criteria
– Rollback strategy should the increment go astray

23. How old are your core systems?
ⓘ Start presenting to display the poll results on this slide.

24. &
25
Changes with future in mind
... simply said: that assumption is absolutely wrong
 new systems need to be continuously modernised too
… unless you want to have the same problem every few years
"We are going to have a new system,
therefore no further investments."
 Fit, Value, Agility  Cost, Complexity, Risk
Year 3 Year 5

25. &
MANTA

26. &
27
Why Data Lineage

27. &
28
How MANTA (and Lineage) Fits

28. &
29
› Driver: Teradata to Snowflake migration to optimize cost and flexibility of
development. Stable reports remain in the on-prem Teradata;
experimental/dev reports in Snowflake (easier scalability as needed).
› Approach: phased migration; metadata driven approach; automated
and repeatable tests. Deployment every other weekend.
› Result: Dependency analysis by MANTA reduced time for
implementation, testing costs, reduced number of defects, allowed for
partitioning of the environment into phases
› Financial institution ($100B +)
Modernization case study

29. &
Modernization Case Study - Target Solution
Teradata Snowflake
Teradata
explore
productionize
Current
Architecture
To-be
Architecture

30. &
31
How Data Lineage Helps in the Process?
› System dependencies
› Assessment of
complexity
› Accurate planning
› Detailed analysis of
transformation code
› Planning individual
modernization phases
› Future system changes
based on metadata
› Augment other processes
(DQ, DataOps, DG, …)
› Phased approach
› Changes in legacy during
the modernization
› Compare before/after
› Automation based on
metadata via API
› Preparing test data
› Reduce need for testing

31. &
32
Where does the data come from?
› Which ETLs need to be adjusted
› What new ETLs need to be developed?
Who consumes the data
› Who will need to approve the changes
› What ETLS need to be changed/developed?
› Other targets that need to be adjusted?
Dependencies Outside of a System

32. &
33
System Internal Dependencies
Goal
› Accurate planning to prevent scope creep & unexpected extra effort
› Define migration phases
Approach
› Understand how the system works – in detail(!)
› See data pipelines for individual attributes
› Understand the dependencies and sequencing

33. &
34
Modernizing = Review the Current Use
Goal:
› Lower migration, maintenance costs & improve manageability
Approach:
› Identify objects no longer in use
› Identify duplicated data pipelines / datasets
› Skip / Optimize during modernization

34. &
35
Planning and Estimating
Goal
› accurate estimates prevent budget and project timeline overruns
Approach
› Understand how complicated the system really is?
› Identify patterns used
› Define approach for migration of specific patterns

35. &
36
Assessing Complexity
Goal
› Data based metric to assess complexity to define approach and resources
needed
Approach:
› Calculated metric defining system complexity
› Transformation complexity
& migration approach
– Simple (40-60%) -> Metadata driven
– Medium (20-35%) -> Pattern-based
– Complex (2-10%) -> Hand Coded

36. &
37
Optimize Testing
Goal:
› Reduce manual testing
› Reduce number of testing rounds
Approach
› Enable developers to smoke test their code
› Compare legacy / new pipeline
› Explain differences (list of exceptions tied to change requests)
› Do this in an automated fashion!

37. &
38
Preparing Test Data*
Goal:
› Enable testers by preparing consistent test environment
› Make testing repeatable
Approach
› Automatically create test sandboxes
› Use lineage information to understand data source
› Limited dataset to a specific use case
* Not implemented as part of this modernization project; comes from a different customer

38. &
39
Goal
› Controlled way to accommodate changes
in legacy environment made during the migration process
Approach:
› Identification of the changes
› Fast analysis of the newly coming changes/fixes
› Automated notifications/reporting
Changes During Migration

39. &
40
Ready for the Future!
Status?
› Both new and legacy environments with documented dataflows
Benefits
› Ability to make future changes efficiently
› Use metadata to augment DQ, DataOps, Incident resolution, …
Snowflake
Teradata
explore
productionize

40. &
Summary

41. &
42
Summary
› Understanding your data dependencies while modernising is crucial
– estimates, risk mitigation, reliable migration
› Both business and IT questions have to be resolved properly
– complex and objective view on the challenge
› Automate data analysis from the day #1
– repeatable, testable, reversible steps
› Profinit Modernisation Framework & MANTA
make modernisation easier

42. Audience Q&A Session
ⓘ Start presenting to display the audience questions on this slide.

43. &
Thank you for your attention

44. &
Backup

45. &
Profinit EU, s.r.o.
Tychonova 2, 160 00 Prague 6 | Phone + 420 224 316 016
Web
www.profinit.eu
LinkedIn
linkedin.com/company/profinit
Twitter
twitter.com/Profinit_EU
Facebook
facebook.com/Profinit.EU
Youtube
Profinit EU
Thank you
for your attention