Crédit Agricole : la Suite Elastic pour des applications bancaires

1. 1
Mohamed Mehdi BEN AISSA
Infrastructure Technical Owner, Credit Agricole Group
Infrastructure Platform (CA-GIP)
23 January 2020
Credit Agricole: Powering
Banking Apps with the
Elastic Stack

2. 2
Speaker
Mohamed Mehdi BEN AISSA
Big Data Technical Architect at CA-GIP
Big Data Infrastructure Technical Owner for CA-CIB

3. 3
Use Cases and Service
Level Agreements (SLAs)

4. 4
CA-GIP & CACIB
Introduction
Elastic at
CA-GIP & CA-CIB
12
Infrastructure B&R
Big Data & Elastic Experts
Big Data & Elastic
Run Team
Big Data & Elastic
Build Team
Elastic
Storage
300 TB
2019 80% 8 000
CA Group
Infrastructure
Collaborators Sites in FranceCreation
Date
17
1 600
CollaboratorsThe world's n°13
bank *
13 36
Locations
around World
* In 2017, measured by Tier One Capital
2TB of Memory
100 Cores

5. 5
Functional Use cases
Risk
Management
Decision
Making
Cash
Management
Regulations

6. 6
Focus on primary use cases
Risk Management/ Regulation
• Aims to replace the current market risk eco-system and phase out the legacy system
(over 10 applications to decommission) to provide the bank with a golden source on
deal & risk indicators across business lines and worldwide
• Addresses ongoing and future regulations (LBF/Volker rules, FRTB, BCBS239, Initial
Margin, Stress EBA/AQR …)
• 3PB of Data on Production to date
Cash Management Transformation
• Strategic program for CA-CIB new business
• Real time Transaction Processing
• Redesign the SI payment for CACIB and international deployment
• Target : 800 millions transactions/day (8 TB/day)
Decisional
Transactional

7. 7
Disaster Recovery
Resiliency
Service Availability
24/24 7/7
Zero Data Loss
Security
Authentication
Authorization
Data Protection
Audit
Performance
Adapted Hardware
OS Optimization
Performance Tuning
Advanced Architecture
Service Level Agreements

8. 8
Security
Authentication
Authorization
Data Protection
Audit
Performance
Adapted Hardware
OS Optimization
Performance Tuning
Advanced Architecture
Service Level Agreements
Disaster Recovery
High Availability Architecture
Data Replication
Multi-DCs Deployment
Rack and Zone Awareness

9. 9
Disaster Recovery
Resiliency
Service Availability
24/24 7/7
Zero Data Loss
Security
Authentication
Authorization
Data Protection
Audit
Performance
Adapted Hardware
OS Optimization
Performance Tuning
Advanced Architecture
Service Level Agreements

10. 10
Disaster Recovery
Resiliency
Service Availability
24/24 7/7
Zero Data Loss
Security
LDAP Integration and SAML
Role Management via Kibana
SSL, TLS and HTTPS
Auditing Security
Performance
Adapted Hardware
OS Optimization
Performance Tuning
Advanced Architecture
Service Level Agreements

11. 11
Disaster Recovery
Resiliency
Service Availability
24/24 7/7
Zero Data Loss
Security
Authentication
Authorization
Data Protection
Audit
Performance
Adapted Hardware
OS Optimization
Performance Tuning
Advanced Architecture
Service Level Agreements

12. 12
Disaster Recovery
Resiliency
Service Availability
24/24 7/7
Zero Data Loss
Security
Authentication
Authorization
Data Protection
Audit
Performance
On-prem / IaaS / CaaS
OS Recommendations
ES Rally
Platinum Support
Service Level Agreements

13. 13
Service Offer Architecture

14. 14
VisualizationIndexingCollect
Bare-Metal & IaaS
Transformation
DATA SOURCES
App 1
App 2
…
App n
Records
Documents
Files
Messages
Streams
HADeploy EZCluster
APPLICATIONS
Ansible
Monitoring Backup DRP
Network Security System Storage
Existing Architecture

15. 15
VisualizationIndexingCollect Transformation
DATA SOURCES
App 1
App 2
…
App n
Records
Documents
Files
Messages
Streams
HADeploy EZCluster
APPLICATIONS
Ansible
Monitoring Backup DRP
Elastic Cloud Entreprise Elastic Cloud on Kube
Bare-Metal +IaaS + CaaS (Kubernetes)
Target Architecture

16. 16
Service Offer
Primary Features

17. 17
End to end monitoring
Infrastructure Monitoring
Full infrastructure monitoring including system, file
system and hardware layers
Applicative Monitoring
Supporting different technologies for databases and
middleware technologies
Functional Monitoring
High level monitoring including functional reporting
and dashboards
Application Performance Management APM
instruments applications to ship performance metrics
to Elasticsearch for visualization in Kibana
HERATBEAT METRICBEAT PACKETBEAT
APM JOLOKIA
MONITORING
TOOLS
INDEXINGANDVIZCOLLECT
LOGSTASH
TRANSFORAMTATION
KIBANA
ELASTICSEARCH

18. 18
Application Performance Management
Architecture
APM Server
APM
Agents
Elasticsearch Kibana
Tomcat
JBOSS
WebSphere
Automatic Kibana integration with
auto-generated dashboards
Different integration mode :
• Embedded (manual setup)
• Agent (automatic setup)
• API
=> Automatic integration with
Ansible and in-house tools

19. 19
Log management
One Service Offer for different needs supporting :
Big Data
Integration with Big Data technologies: Hadoop,
Spark, Kafka, HBase, etc.
Grid Computing
Parsing and indexing a high data volume of logs
generated by thousands of cores.
Kubernetes
Log management for Kubernetes infrastructure and
managed containers and pods.
And …
The support of all used technologies at CA-CIB with
automatic logs integration and visualization
APACHE
KAFKA
FILEBEAT LOG4J WINLOGBEAT
INDEXINGANDVIZCOLLECT
LOGSTASH
TRANSFORAMTATION
KIBANA
ELASTICSEARCH

20. 20
Functional Use Cases &
Applicative architectures

21. 21
Big Data Integration
Risk
Management
Decision
Making
Cash
Management
Regulations

22. 22
Big Data Integration
Cash Management
Distributed Message Bus
Kafka Broker
Kafka Broker
Kafka Broker
Event Processing
Micro-Batch
Processing
Distributed Processing
State
Backend
Serving
Layer
Historic
Storage
Distributed Persistent Layers
Transactions
Cascade Mode (more than one iteration)
Distributed End-to-end
Tracing for EOS

23. 23
GRID Integration
Risk
Management
Decision
Making
Cash
Management
Regulations

24. 24
GRID Integration
Market Risk / Regulation
Distributed Message Bus
Kafka Broker
Kafka Broker
Kafka Broker
GRID Computing
Server 1
Server 2
Server 3
Server N
Micro-Batch
Processing
Distributed Processing
Elasticsearch Kibana
logs
Distributed Log
Management
More than 500 GB of logs generated
per day by thousands cores

25. 25
Security Audit
Risk
Management
Decision
Making
Cash
Management
Regulations

26. 26
Security Audit
Architecture
Logstash Elasticsearch Kibana
Audit CollectSupported Technologies
Apache Ranger
AuditBeat
Big Data Clusters
Databases
Servers
Audit Indexing and Reporting (Hot Data)
Security Alerting, reporting
and Analytics

27. 27
CONCLUSION

28. 28
Conclusion
Key Success Factors
• Support and correlation of different features
• Respect of the Service Level Agreements
• Multi-tenant Platform supporting several
functional Use Cases
Perspective
• More Agility with ECE and ECK

29. 29
THANK FOR YOUR
ATTENTION