Synopsis: Modern enterprises anticipate business requirements and work proactively to optimise the outcomes. If they don’t renovate or reinvent their data architectures, they lose customers, and market share. So my talk will be in detailing the importance of data architecture, architectural challenges if is not addressed and a case study - the learnings and success story by fixing the issues at the root - at the data storage & access. Target Audience: Principal Software engineers & Architects Key Takeaways: Importance of Modern Data Architecture, PostgreSQL & JSONB I have given a talk @ https://hasgeek.com/rootconf/elasticsearch-users-meetup-hyderabad/

1. Data Architecture for Modern Enterprises
Presented @ hasgeek-rootconf-elasticsearch-users-meetup-hyderabad, Nov 2019

2. Hyderabad

3. AgendaData - The Core Why Data Architecture?
Business Use Case
Lessons Learnt
The Journey & Success Story

4. Data - The Domain
Films Character ActorDirector Drama Comedy Thriller
Patient Doctor Operation Disease Pills Visit
Video
Health Care
Symptom
Rom
Nurs

5. Data - Modeling Techniques
20XX
NoSQL
1990s
Object Oriented
1970s
Relational
Navigational
1960s

6. The Data
HBR's Alarming Survey Results
survey participants: 64 c-level technology and business executives representing
large corporations such as American Express, Ford Motor, General Electric,
General Motors, and Johnson & Johnson
2019 Big Data and AI Executive Survey

7. The Data Architecture
Data architecture describes how data is collected, stored,
transformed, distributed, and consumed
- Wikipedia

8. Data Architecture - A (Legacy) Sample
More
Data Sources
3rd Party APIs
Databases,
DWH & Lakes
Different Files
NoSQL
Application (or) Business Processing Data Consumers
More
Data Visualisers
3rd Party Apps
In House Apps
Files
Chaos
Calls for Renovation/Reinvention

9. Message Bus
Data Consumers
More
Data Visualisers
ML Tools
Applications
Microservices
More
Data Sources
3rd Party APIs
Databases,
DWH & Lakes
Different Files
NoSQL
Data Architecture - A (Modern) Sample
Application (or) Business Processing

10. Core Enablers of Modern Business
Solid Data Architecture - The Why?
IoT AIBig Data Cloud Native ML
Robust Data Architecture

11. Client Portfolio
Business Use Case
Through-Channel Marketing Automation (TCMA)
SaaS platform
➔ Accelerates Brand sales in Local Market
➔ Shares comprehensive Analytics on Individual
Tactics & Campaigns
➔ Helps Brands to adapt to market change
based on data driven insights

12. The (Domain) Data
Business Use Case
CampaignsRetailers Target Audience
Order Snapshots
Brands
Orders
and so on...

13. The Data Analysis
Business Use Case
The target Audience, the orders, the order
snapshots all capture different data points
per group of records
Chess List
➔ Classic FIDE
➔ Blitz FIDE
➔ Rapid FIDE
Football List
➔ Height
➔ Weight
➔ Nationality
Wrestling List
➔ Ring Name
➔ Height
➔ Weight
➔ Debut

14. Tech Stack
Business Use Case

15. R1_Orders R1_Snapshots R1_XYZ Indices
R1_Audience
_1
R1_Audience
_2
R1_Audience
_3
Indices
R1_Audience
_4
R1_Audience
_5
R1_Audience
_N
Indices
R2_Orders R2_Snapshots R2_XYZ Indices
R2_Audience
_1
R2_Audience
_2
R2_Audience
_3
Indices
R2_Audience
_4
R2_Audience
_5
R2_Audience
_N
Indices
R3_Orders R3_Snapshots R3_XYZ Indices
R3_Audience
_1
R3_Audience
_2
R3_Audience
_3
Indices
R3_Audience
_4
R3_Audience
_5
R3_Audience
_N
Indices
RN_Orders RN_Snapshots RN_XYZ Indices
RN_Audience
_1
RN_Audience
_2
RN_Audience
_3
Indices
RN_Audience
_4
RN_Audience
_5
RN_Audience
_N
Indices
450K Tables for 4K Customers -
Scalable?
Chaos.. Calls for Renovation!!
The Challenges
Business Use Case

16. The Reinvention
Business Use Case
Microservices
Netflix has fixed the Scalability Issues by adopting
microservices architecture. So, why not us?
NoSQL
No need to create multiple tables to support dynamic columns
instead, let us leverage NoSQL schema-less approach
By Client’s Labs Team
Is the foundation (data architecture) robust?

17. A Business Unit of
The solution
Business Use Case
proposed by
Cloud Native architecture with java microservices
Phase#2: Reinvention
Hybrid Data Architecture - Schema & Schemaless approach
Phase#1: Renovation

18. Renovation Highlights
Business Use Case
BRAND LIST FIRST_NAME LAST_NAME CUSTOM_DATA
B1 CHESS Vishwanathan Anand {“blitz”: 2786, “rapid”:2737}
B1 CHESS Garry Kasparov {“classic”: 2812, “blitz”: 2801, “rapid”:2783}
B2 FOOTBALL Christiano Ronaldo {“nationality”: “Portugal”, “height”: “1.87M”}
B2 FOOTBALL Lionel Messi {“nationality”: “Argentina”, “height”: “1.7M”}
B3 WRESTLING Geetha Phogat {“coach”: “Mahavir Singh”, “height”:”1.62M”}
#1 No Multiple Tables & Indices - Dynamic Columns using JSONB & GIN

19. #2 Partitioning
Renovation Highlights continues...
Business Use Case

20. A Business Unit of
Renovation Benefits
Business Use Case
proposed byNo Tables & Indices on the fly
➔ Instead Data was Organised for effective Storage
& quick Access
➔ Able to accomodate more Brands with the same
infrastructure
#2: Maximum Scalability
➔ The data architecture issues were fixed
➔ No New Tech Stack introduction, so not much
learning curve & additional deployment effort
#1: Minimal Disruption

21. Renovation Architecture
Verification Phase
Business Use Case
LIST T-SHIRT AUDIENCE
1 S 1,000
2 M 10,000
3 L 100,000
4 XL 1,000,000

22. All reads were < 2 seconds even for the 100K audience list
The Proof
Business Use Case

23. Recap
The Fundamentals
Data is an Asset
Rights to get Repaired
Organised
Consumer Focused
Single Source of Truth

24. Thank You!
Kayalvizhi Noorul Ameen, Principal Architect @ Pramati Technologies
Image Credit: unsplash.com & flaticon.com
The Case Study - published in Imaginea