www.LnTInfotech.com
DI Modernization at the Age of
Digital Disruptions
Amar Roy
L&T Infotech Ltd.
2
About Myself
I help my customers in their digital journey by developing
their big, small & fast data solution.
3
About L&T Infotech
 L&T Infotech is a global IT services and solutions
provider. We provide the winning edge to our clients by
leveraging our Business-to-IT Connect and Deeply
Committed People.
 Our clients have found in us a right-size partner who
combines scale, stability and customer-centricity.
4
Digital businesses - disrupting “business as usual”!
5
Data is the glue of all digital businesses!
And an efficient data integration platform is the logical
starting point towards success
6
Today we will discuss…
 What is a digital business?
 Needs and challenges of DI modernization
 Strategies an organization can adopt
 Capabilities to be acquired
 Platform & toolset
 and Operating model to make the transformation to work
7
What does it mean to be Digital?
8
Being Digital Means*…
And it demands innovation, speed & agility!
 creating value at the new frontiers of the business world
 creating value in the processes that execute a vision of
customer experiences
 and building foundational capabilities that support the
entire structure.
*Source: What ‘digital’ really means | McKinsey & Company
9
This is new normal of
customer service and
satisfaction!
Uber runs its business
on a platform which
uses enables this level
of customer service
and more…
10
Image Source: http://www.datanami.com/2015/10/05/how-uber-uses-spark-and-hadoop-to-optimize-customer-experience/
It’s fundamentally a data problem!
- Aaron Schildkrout, Head of Data @ Uber
11
However, this is not easy for every organizations!
12
Their present Data Integration landscape looks like
this!
Where to start?
13
Clearly , it is not only about technology!
14
You will need to
choose the right modernization strategy,
acquire the right capabilities and integrate with
existing ones,
implement state of the art platform & toolset,
adopt to an agile operating model which makes
everything work seamlessly
to get your DI ready for Digital Transformation…
15
What are the right strategy an organizations can adopt?
Depending upon your situation, you can choose either of
these 2 strategies,
Big Bang Approach
2-Speed Transformation
Most of our customer adopts the 2nd strategy
16
 Identify the candidates which
need to change at faster speed
 Not all DI applications need to
change at faster speed
17
Back office adopts slowly
18
Customer facing systems needs to change faster!
19
Big Data is only a piece in the DI puzzle…
Stream
Computing will
make real-time
integration
possible
Cloud
Infrastructure
will provide one-
click provisioning
and elasticity
API Based
Integration will
enable a data as
Microservices
Logical Data
Warehouse
will contain
Small/Big &
Fast Data
20
API based integration provides seamless cross channel
experience by enabling data as a microservice…
 design of data services
around business
capabilities, automated
deployment and
decentralized control of
language & data.
 REST is the de facto
standard
Image Source: https://blog.karmawifi.com/how-we-build-microservices-at-karma-71497a89bfb4#.ixbgljju0
21
Real time streaming for responsiveness…
Image Source: https://spark-summit.org/2014/wp-content/.../Lambda-Architecture-Jim-Scott..pdf
22
Logical DW extends existing infrastructure to cater new data and related workloads…
Data
Quality
ILM
Data Governance
Data
Integration
MDM
Source
Security
Web
Services
APIs
SQL
Queries
Search
Custom
Applications
ERP
SCM
CRM
Image
Audio
& Video
Machine
Logs
Emails
Web &
Social
Sensor
Data
Documents
Eg Pdf,
Docs
StructuredUn-structured
Data Acquisition Layer Data Consumption Layer
Marketing
Executives
Operational
Managers
Data Scientists
Business
Analysts
Engineers
Customers
Partners
Frontline
Workers
Data
Lake Archived
Data
Master
Data
Staging
Data
Non Prod
Data
Metadata Management
Data
Virtualization
Analytical
Data
Data Fill
Platform
Data Mill Platform
Data Drill Platform
Data Warehousing Platform
Near Real Time
Load
Batch
Load
Advanced
Analytics
 Prescriptive
 Predictive
 Descriptive
Business
Intelligence
 BI Reports
 Ana Reports
 Dashboards
 Visualization
OLAPODS Data Marts
23
One-click provisioning & elasticity through a cloud
infrastructure; allowing faster innovation
“…Cloud Computing is
multi-tenant, it’s faster,
half the cost, pay as
you go, it grows as you
grow or shrinks as you
shrink. It is extremely
efficient…”
- Marc Benioff, CEO SFDC [Oracle
Open World 2010]
24
Plethora of tools to architect your agile & scalable data platform
Data
Sources
Flow
Controller
Stream
Processor
Storage Analytics
 Enterprise
Applications
 Mobile Devices
 IoT Sensors
 System logs
 Web Scrapping
 Messaging
Queue i.e.
Kafka, Tibco
EMS
 Amazon
Kinesys
 Storm
 Spark
Streaming
 Informatica
CDC
 Flink
 Azure
Streaming
 RDBMS
 NoSQL
 HDFS
 Cloud Storage
S3, HDInsight
 Cloud DB -
Redshift
 Custom
Application –
D3
 Tools –
Tableau, Qlik
 Elastic Search
 Many more
25
Because of the speed and agility digital initiatives will usher
a cultural shift in the way organization builds and deliver
software…
26
DevOps & Continuous Integration – Lean Execution Model for
DI TransformationStart
Continuous
Integration Server
Source
code
Repositor
y
Resolve
Dependenci
es
Get Latest
job/packag
e
Integration
of
components
Static
Analysis
Deploy in
QA
Environme
nt
Unit &
Functional Test
Regression
TestResult
Validation
Update
Matching
Defects
Send
Notificatio
n
(Start)
(End)
Build Tool
(xml test
cases)
(Coding Standard,
Sanity checks)
(xml test
cases)
Java API
Trigger
Mechanis
m
Code
Check-in
Matrix
Dashboar
d
(pearl, shell, js)
Test Management
Server
SMTP
Server
Bug
Reporting
Tool
Automated Test
Script
Raise
Defect
Feedback
ETL jobs
BI
Dashboard
s
27
“Companies rarely die from moving too fast, and they
frequently die from moving too slowly.”
- Reed Hastings, CEO of Netflix
www.LnTInfotech.com
Please send your questions to amar.roy@lntinfotech.com
Let’s Solve

Di in the age of digital disruptions v1.0

  • 1.
    www.LnTInfotech.com DI Modernization atthe Age of Digital Disruptions Amar Roy L&T Infotech Ltd.
  • 2.
    2 About Myself I helpmy customers in their digital journey by developing their big, small & fast data solution.
  • 3.
    3 About L&T Infotech L&T Infotech is a global IT services and solutions provider. We provide the winning edge to our clients by leveraging our Business-to-IT Connect and Deeply Committed People.  Our clients have found in us a right-size partner who combines scale, stability and customer-centricity.
  • 4.
    4 Digital businesses -disrupting “business as usual”!
  • 5.
    5 Data is theglue of all digital businesses! And an efficient data integration platform is the logical starting point towards success
  • 6.
    6 Today we willdiscuss…  What is a digital business?  Needs and challenges of DI modernization  Strategies an organization can adopt  Capabilities to be acquired  Platform & toolset  and Operating model to make the transformation to work
  • 7.
    7 What does itmean to be Digital?
  • 8.
    8 Being Digital Means*… Andit demands innovation, speed & agility!  creating value at the new frontiers of the business world  creating value in the processes that execute a vision of customer experiences  and building foundational capabilities that support the entire structure. *Source: What ‘digital’ really means | McKinsey & Company
  • 9.
    9 This is newnormal of customer service and satisfaction! Uber runs its business on a platform which uses enables this level of customer service and more…
  • 10.
  • 11.
    11 However, this isnot easy for every organizations!
  • 12.
    12 Their present DataIntegration landscape looks like this! Where to start?
  • 13.
    13 Clearly , itis not only about technology!
  • 14.
    14 You will needto choose the right modernization strategy, acquire the right capabilities and integrate with existing ones, implement state of the art platform & toolset, adopt to an agile operating model which makes everything work seamlessly to get your DI ready for Digital Transformation…
  • 15.
    15 What are theright strategy an organizations can adopt? Depending upon your situation, you can choose either of these 2 strategies, Big Bang Approach 2-Speed Transformation Most of our customer adopts the 2nd strategy
  • 16.
    16  Identify thecandidates which need to change at faster speed  Not all DI applications need to change at faster speed
  • 17.
  • 18.
    18 Customer facing systemsneeds to change faster!
  • 19.
    19 Big Data isonly a piece in the DI puzzle… Stream Computing will make real-time integration possible Cloud Infrastructure will provide one- click provisioning and elasticity API Based Integration will enable a data as Microservices Logical Data Warehouse will contain Small/Big & Fast Data
  • 20.
    20 API based integrationprovides seamless cross channel experience by enabling data as a microservice…  design of data services around business capabilities, automated deployment and decentralized control of language & data.  REST is the de facto standard Image Source: https://blog.karmawifi.com/how-we-build-microservices-at-karma-71497a89bfb4#.ixbgljju0
  • 21.
    21 Real time streamingfor responsiveness… Image Source: https://spark-summit.org/2014/wp-content/.../Lambda-Architecture-Jim-Scott..pdf
  • 22.
    22 Logical DW extendsexisting infrastructure to cater new data and related workloads… Data Quality ILM Data Governance Data Integration MDM Source Security Web Services APIs SQL Queries Search Custom Applications ERP SCM CRM Image Audio & Video Machine Logs Emails Web & Social Sensor Data Documents Eg Pdf, Docs StructuredUn-structured Data Acquisition Layer Data Consumption Layer Marketing Executives Operational Managers Data Scientists Business Analysts Engineers Customers Partners Frontline Workers Data Lake Archived Data Master Data Staging Data Non Prod Data Metadata Management Data Virtualization Analytical Data Data Fill Platform Data Mill Platform Data Drill Platform Data Warehousing Platform Near Real Time Load Batch Load Advanced Analytics  Prescriptive  Predictive  Descriptive Business Intelligence  BI Reports  Ana Reports  Dashboards  Visualization OLAPODS Data Marts
  • 23.
    23 One-click provisioning &elasticity through a cloud infrastructure; allowing faster innovation “…Cloud Computing is multi-tenant, it’s faster, half the cost, pay as you go, it grows as you grow or shrinks as you shrink. It is extremely efficient…” - Marc Benioff, CEO SFDC [Oracle Open World 2010]
  • 24.
    24 Plethora of toolsto architect your agile & scalable data platform Data Sources Flow Controller Stream Processor Storage Analytics  Enterprise Applications  Mobile Devices  IoT Sensors  System logs  Web Scrapping  Messaging Queue i.e. Kafka, Tibco EMS  Amazon Kinesys  Storm  Spark Streaming  Informatica CDC  Flink  Azure Streaming  RDBMS  NoSQL  HDFS  Cloud Storage S3, HDInsight  Cloud DB - Redshift  Custom Application – D3  Tools – Tableau, Qlik  Elastic Search  Many more
  • 25.
    25 Because of thespeed and agility digital initiatives will usher a cultural shift in the way organization builds and deliver software…
  • 26.
    26 DevOps & ContinuousIntegration – Lean Execution Model for DI TransformationStart Continuous Integration Server Source code Repositor y Resolve Dependenci es Get Latest job/packag e Integration of components Static Analysis Deploy in QA Environme nt Unit & Functional Test Regression TestResult Validation Update Matching Defects Send Notificatio n (Start) (End) Build Tool (xml test cases) (Coding Standard, Sanity checks) (xml test cases) Java API Trigger Mechanis m Code Check-in Matrix Dashboar d (pearl, shell, js) Test Management Server SMTP Server Bug Reporting Tool Automated Test Script Raise Defect Feedback ETL jobs BI Dashboard s
  • 27.
    27 “Companies rarely diefrom moving too fast, and they frequently die from moving too slowly.” - Reed Hastings, CEO of Netflix
  • 28.
    www.LnTInfotech.com Please send yourquestions to amar.roy@lntinfotech.com Let’s Solve

Editor's Notes

  • #3 Good afternoon everyone! My name is Amar Roy and I am a principal solution architect at L&T Infotech. I help my customers in their digital journey by developing their big, small & fast data solution.
  • #4 Among other service lines, we have keen focus in Analytics and Information management in the Context of digital transformation of our customers.
  • #5 Let’s get started with today’s topic, but before we get their I have a question. What is common among these companies?   Yes, these are all digital businesses; but they are digital disruptors as well.   Today digital businesses are everywhere. Every day they are not only developing new business models but disrupting those of the incumbents. Amazon did this to Border’s brick and mortar bookstores. Uber is virtually wiping out the traditional Taxi industry by allowing people to provide taxi service if they have a smart phone and a car. And they are coming out in all segments of businesses. So in a way posing threat to the incumbents with the data centric innovation & speed. So clearly there is a need of change to be in business and be competitive!
  • #6 All these digital companies have one more thing in common; they are master in using data for running their business. They use data for product innovation, customer insight, process optimization and speed. In fact , data is used as Glue of all digital businesses. And before any bit of data can be used, it must be ingested, processed and presented to to internal/external consumers. This is the reason DI plays the central role in the digital journey of an Organization.
  • #7 My goal of this presentation, is to elaborate a clear framework for DI modernization which can sustain the digital initiative of a company…
  • #8 But even before we discuss the importance of data integration modernization in the context of digital disruption, let’s understand what it means to be digital. And understand why it is disruptive in nature.
  • #10 Last year, I took an Uber from my office in Edison to Newark Airport. When I got down at Newark, I looked at my Uber app and Realized the driver did not closed the trip immediately! He drove another 2-3 miles before he closed the trip. I was literally pissed and gave a bad rating with description of what happened. Within next 6 min, I got this email… This is the benchmark digital companies are setting… Digital businesses are setting new benchmarks for speed, agility, and user-friendliness, consumers expect similar online performance from banks, retailers, and telecommunications companies.
  • #11 Before we move on to how DI modernization can be done for Digital Age, lets pause for a moment to examine what kind of systems and infrastructure enables Uber to provide such un parallel customer service with speed and Agility. As expected, Uber Head of Data, Aaron Schildkrout says “It’s fundamentally a data problem!” It is evident from this architecture , Uber has focused on building a robust DI architecture which enable processing huge amount of data real time. This is the core of their business…
  • #12 But unlike the Uber, incumbent organizations does not have the luxury of starting with a clean slate. Their Data Integration landscape has grown overly complex over time and can’t be changed overnight.
  • #13 So what do they do if their DI systems looks like this? Where should they start to make their DI more agile and responsive without disturbing all functions at once?
  • #14 So technology can’t alone solve this problem. They need a more holistic approach for a successful DI modernization.
  • #15 This transformation is possible if these 4 activities executed well   choose the right modernization strategy, acquire the right capabilities and integrate with existing ones, implement state of the art platform & toolset, adopt to an agile operating model which makes everything work seamlessly   We will examine these in next few slides and understand how these will facilitate the DI modernization in the digital journey of the organization.
  • #16 We have found there are two ways organization can make this happen Big Bang Approach – Change the entire DI landscape as part of multi year transformation program. However, most often that is not practical 2-Speed Transformation – In today’s competitive environment, you need to adopt an incremental approach which McKinsey termed as two – speed approach. Most of our customer adopts the 2nd strategy. One of our Oil & Gas customer actually followed the approach #1 But one large US bank has chosen to go for approach #2
  • #17 In an organization not all DI application need to change at the same speed or to be touched at all during DI modernization initiative. To start with, create a distinction between these two set of applications and build a plan on how these two set of applications will be handled in your DI transformation program.
  • #18 Core transactional system of records needs to be stable & consistent for various reasons Like risk & regulatory compliance etc. These are typically your ERP, Financial and Inventory management systems. They need Accuracy over speed. And should not be changed often. We can leave them alone if they are doing what they are expected to do…
  • #19 a. However, your customer facing systems needs to change fast and often multiple time within a day. So these should be your primary focus for the DI transformation project. b. These systems needs to be agile and change often. Think about major retailer like Amazon, Media and Entertainment Company like HBO/Netflix. They changes often – even multiple time during the same day… c. These application should be focused during the DI transformation…
  • #20 a. There is a notion that Digital is all about big data analytics. Not completely true! To provide the speed & agility data integration part is equally important as storing And analyzing big data. You need to make your DI platform capable of handling big, small & fast data. We have seen these are the broad DI capabilities which will make the customer facing systems, Provide seamless experience across channels Real-time insight on customer behavior Zero down time Auto-scale up/down based on load
  • #21 Amazon and Netflix are among the biggest users of Microservices. Amazon has more than 100 micro-services in their site. Because of the modular nature micro-services enables automated deployment with zero down time of business Critical websites. You can deploy multiple changes within normal business hours with out disrupting your Customer experience. Most important from DI perspective, you can reduce the need to move data across systems and save valuable ETL development time. Finally you, can virtually remove the nightly FTP jobs , make the data available on-demand freeing up the Network traffic.
  • #22 Real-time streaming data integration is the other component of speed, agility and customer experience. We have seen the mail that Uber customer support sent within 6 min of submitting my grievance! Traditionally , all data integration were designed for processing batches. Even, in HDFS in essence a Batch processing system with huge throughput but high latency. That was good, but today most of your data comes in streams, be it sensor data or tweets from customer. So , besides batch processing you need the ability of processing data streams as well. Lambda Architecture , is an architecture style which implements both batch as well as stream processing Side by side. You can see the technology components of such an architecture. Finally, you need a database Which will provide the data to applications with extremely low latency
  • #23 However, most of the organizations has invested heavily on the Enterprise Data Warehouse ecosystem. Most often, they need to be expanded to include new data sources or new types of data. This gives rise To what we call is Logical Data Warehouse. There is another school of thought which calls for a Hadoop DataLake. But we have seen mixed success Hadoop Data Lake.
  • #24 In a digital business workload varies , often unexpectedly. Organizations should look for an elastic infrastructure which expands and contracts with load. And cloud infrastructure Provides exactly the same. If you see , most of the digital businesses born in cloud and it has given them extra-ordinary scale With minimal investment. They can provision a cluster within minute, ingest data and try out an Idea. They can fail fast and innovate faster… In a two speed DI transformation, cloud infrastructure is ideal for the fast moving applications.
  • #25 For your data pipeline you have lots of choice from open source or Informatica to choose from
  • #26 Digital businesses breaks silos within organizations…It brings level of transparency in the way software is built, tested and deployed. Most often software development is tied up with product launch and success of an initiative. So these often draws more attention of Senior executives. On the team front, project mindset gets dissolved paving the way for product mindset. A team of competent engineers owns and Improves the code base. People is rewarded for failing faster and innovation…
  • #27 While the system of records follow waterfall release methodology, customer facing systems needs to be agile. Changes to be deployed in days not hours. So DevOps and Continuous Delivery is the way forward. Automated build solution Automated Deployment Solution Automated testing solution – Test driven development Capture metrics and improve