Financial companies need Java EE to power its business today. Rakuten Card, one of the largest credit card companies in Japan, adopted Java EE 7 for its credit card core systems rearchitecture, from one of the oldest COBOL based mainframe in Japan. Additionally, we chose Apache Spark for super rapid batch execution platform. We completed this big core system migration project successfully. You can learn why we choose Java EE, and Apache Spark for super rapid batch execution, and our experiences and lessons we learned. How to start such a big project? Why we choose it, how we ported, how use Apache Spark for performance improvements, and launched with? We’ll answer these questions and any that you may have.

1. Java EE 7 with Apache Spark for the World’s
Largest Credit Card Core Systems
[CON4998]
Oct 4, 2017
Hirofumi Iwasaki
Ville Misaki
System Strategy Department,
Rakuten Card Co., Ltd.

2. 2
Speaker Biography
 Hirofumi Iwasaki @HirofumiIwasaki
 Group Manager
 Technology Strategy Group, System Strategy Department,
Rakuten Card Co., Ltd.
 Career
 Planning, designing & implementation of huge enterprise systems for
financial, manufacturing and public systems with Java EE in Japan over
18 years.
 Opus, Lectures, etc.
 Conferences: OOW 2014, JavaOne 2015, 2014, Java Day Tokyo 2014-
2015, Rakuten Tech Conference 2013-2016, etc.

3. 3
Agenda
Part 1 – Perfect Design
1. About Rakuten Card
2. Background
 Hardware
 Software
 Database
Part 2 – Harsh Reliability
3. Performance
4. Apache Spark
5. Judgement Day
6. Into the Future

4. 4
1. About Rakuten Card

5. 5
About Rakuten Group
 Unified brand, ecosystems around the world.

6. FC Barcelona partnership
kicked off on July 1, 2017

7. Warriors and Rakuten
Form Jersey Partnership
in the 2017-18 NBA season

8. 8
About Rakuten Card
 Top-level credit card
company in Japan
 Core of Rakuten eco
systems.
 3rd position of total
transaction volume in 2016.
Growing rapidly.

9. 9
Conference session on JavaOne 2014, 2015
 Shared with web front end
systems improvement
activities.
 Based on Java EE 6
 Started from Glassfish 3,
migrated to WebLogic
server 12c
 In-house development
 Great success

10. 10
2. Background

11. 11
Card processing systems
Core Systems
Web Systems
External Systems
Intra Systems

12. 12
Old core systems - Mainframe
Mainframe
 Old architecture – over 20 years
 High cost structure
 Capacity and performance
limitation – no scale out
 Low maintainability with piled
programs and old architecture
database "NDB"
 Risk against vendor locked-in
 Limitation of the security for the
significant data

13. 13
Limitation of old mainframe systems – Areas
Business
Operations
Development

14. 14
Limitation of old mainframe systems – Business
Old New
 Cannot scale-out  Apply scale-out enabled
architecture, with Oracle RAC
and clustered WebLogic server.
 Low connectivity to other
systems
 Apply Java EE and latest
protocol.
 Less security management on
data
 Apply Oracle database
security options.
 No latest auto testing
environment
 Introduce latest auto testing
environment.

15. 15
Limitation of old mainframe systems – Development
Old New
 No local development  Apply Java EE and Oracle DB
for local dev.
 Hard to understand because
of its old architecture
 Apply latest Java EE for its
basement.
 Poor version control systems  Introduce git server and issue
track systems.
 No development community  Apply Java EE and join open
community.

16. 16
Limitation of old mainframe systems – Operation
Old New
 Poor automated operations  Introduce Jenkins and
automations.
 Manual error monitoring  Include Zabbix monitoring to
cover the new core system.
 Difficult to pin-point cause of
error
 Use standard Java tools: stack
traces, Flight Recorder, etc.
 Tons of unused codes  Apply automated source code
analyzing tool.

17. 17
Phase of the improvement – 3.0
1.0
Initial phase
2.0 In-house
development
3.0
Standardization
4.0
Data Optimized
Outsource based,
just started.
Vendor locked-in.
In-house
development,
differentiate with
lower costs and
faster delivery.
Standardized
system
architecture, both
for hardware and
software.
Overwhelming
differentiation,
with enabling
architecture for
customer centric
service.
Achieved Next
Current Standard
Architecture

18. 18
Horizontal expansion from web systems
18
2013 2017
Web systems
Core
Systems
Expand

19. 19
Oracle Exalogic
+ Exadata + ZFS Servers
Big Improvement - Functionality: Hardware 1/2
19
Mainframe
Old New
Core
Systems

20. 20
Big Improvement - Functionality: Hardware 2/2
20
Oracle Exalogic
+ Exadata + ZFS Servers
Oracle Cloud Machine
(On premise private cloud)
For temporarily
request spiking
Low-Cost
Temp
Resource
New
Core
Systems

21. 21
Big Improvement - Reliability: Software Platform
 Financial de-facto standard
 Java EE compliant.
 Matured, from 1997.
 Financial de-facto standard
 ISO/IEC 9075 SQL compliant
 Matured, from 1983.
COBOL
Network
DB
App Server
Database
Old New
WebLogic Server
Oracle Database

22. 22
Big Improvement - Portability: Platform independent
Hardware, OS, app
server independent,
vendor free.
Mainframe,
Japanese COBOL,
vendor locked-in
Old New
Widfly
Payara
WebLogic
hp-ux
AIXSolaris
Linux
Windows
macOS
WebSphere

23. 23
Software Migration – Conversion
Japanese COBOL
Source code
Source code
Custom made
source code
converter
 Convert from Japanese
COBOL to Java EE
 Keep original core
business logic

24. 24
Software Migration – Conversion: Dual Source
From Web Systems,
For New Logic
COBOL
From Old System,
converted to Java
 Ease of migration, resource re-use
 Introduce power of Java EE
 Introduce converter from YPS to Java
“Dual Source Architecture”
Japanese
COBOL
 Japanese source code
 Almost abandoned
 No books, no community
Old New

25. 25
Big Improvement – Efficiency: API-nized
BIG-IP
Real-time Servers
(WebLogic)
Batch Servers
(Spark & Java)
Façade
Rich clients Façade
Façade
Intranet
External
Intra
Exadata
Mail
Form
BIG-IP
Façade
BIG-IP
External
customers
Scheduler
CoreBusinessLogicAPIs
Operation
terminal
Web
browser
Old New

26. 26
New Database
Overview of Data Conversion
ISAM
VSAM
NDB
Java
Business Logics
Japanese COBOL
business logics
Common Module
Data Accessor Common Module
Database Accessor
Migrate
Web
Database
Old New

27. 27
New Database
Schema Conversion Policy – From ISAM/VSAM
- Record Key
(Unique)
- Record Key
(Not Unique)
A_RDB_TABLE
----------------------------
- PRIMARY KEY
- OTHER COLUMN
Add unique
index.
Add index
only.
Old New
ISAM/VSAM

28. 28
Auto testing environment
3. Run tests
on staging environment
2. Execute auto testing
on several times
1. Register auto test scenarios
 Automatic testing
using latest IBM
Rational test software.
 Regression test
enabled when
something changed.
 Reduce error
possibilities on
production release.
Testing
Server

29. 29
Part 2

30. 30
Speaker Biography
 Ville Misaki
 Senior Software Engineer
 Technology Strategy Group,
System Strategy Department,
Rakuten Card Co., Ltd
 Career
 15+ years; 3 years at Rakuten
 In Finland, the Netherlands, Japan
 Java (EE), Perl, C++, web systems, relational
databases, performance optimization & security

31. 31
Agenda
Part 1 – Perfect Design
1. About Rakuten Card
2. Background
 Hardware
 Software
 Database
Part 2 – Harsh Reliability
3. Performance
4. Apache Spark
5. Judgement Day
6. Into the Future

32. 32
3. Performance

33. 33
Performance – First Trial
vs.

34. 34
Performance – First Trial
vs.

35. 35
Performance – First Trial – Details
Start
Slow
Slow
 Batches are run as networks
 Hierarchical
 Critical path
 Time window

36. 36
Bad Performance – Causes
 Automatic code conversion
 COBOL program flow emulated in Java
 COBOL-like data structures in Java
 DB access logic
 Business logic built on network DB
 NDB and RDB are good at different tasks

37. 37
Bad Performance – Cause: COBOL Emulation
 COBOL vs. Java
 Goto statement – imitation is complex
 Sub-program calls – heavy
 No local variables – tight coupling
 No libraries – copy&paste code
 Few shared data structures – copy&paste definition
 No shared enum/constant – magic numbers

38. 38
Bad Performance – Cause: COBOL Emulation
 COBOL data structures
 Fixed length – hard-coded
 String-based
 Data block inside program
 Often thousands of fields
 Hierarchical fields
 Content is joined/split automatically
 Variable namespace under each parent
 Even five levels deep

39. 39
Bad Performance – Cause: COBOL Emulation

40. 40
Bad Performance – Cause: NDB Emulation
 Logic optimized for NDB
 Read sequentially
 Data pre-sorted
 Data pre-formatted
 Emulate in RDB
 Uphill battle
NDB RDB
Search Slow Fast
Sequential Access Fast Slow
Sorting Slow Fast
Formatting Fast Slow

41. 41
Performance – Must Improve
 New system must be faster
 Time until launch:
1 year

42. 42
Performance – Solutions?
 Options?
 Redesign and re-implement from scratch
 Not feasible
 Optimize framework
 Limited effectiveness
 Parallelize batches
 Elastic brute-force

43. 43
4. Apache Spark

44. 44
Performance – Run in Parallel
Time
Sequential
Parallel

45. 45
Apache Spark
Cluster Node
Cluster Node
Cluster Node
Cluster Node
Cluster Node
Cluster Node
Bootstrap
SharedMemory
Scheduler

46. 46
Apache Spark – Challenges
1. Making business logic parallel
 Independent processing
2. I/O
 Data transferred over network
3. Data ordering
 Shuffles

47. 47
Apache Spark – Challenges: Independent Processing
 Problem: input data rows are not independent!
 Red flags
 Fields not initialized for each row
 Code forks early (header & data?)
 Legacy code analysis
 Refactor
 Fields to local variables
 Extract data structures
 Initialize data for each row
 Run & see
321
3
2
1 Reference?

48. 48
Apache Spark – Challenge 1: Independent Processing, Solutions
1. Group related rows together
2. Process header rows separately
3. Modify business logic

49. 49
Apache Spark – Challenge 1: Independent Processing, Solution 1
Group related rows together
 Custom data reader
 Multiple rows behave like one row
 Process each group row in a loop, on
the same node
 Pro
 Business logic not modified
 Con
 Relationships may be too complex
 Groups may grow too big
ID Data
1 …
1 …
2 …
3 …
3 …
4 …

50. 50
Apache Spark – Challenge 1: Independent Processing, Solution 2
Process header rows separately
 Run business logic for header rows first
 Collect result in NavigableMap
 Run business logic for data rows
 Initialize data from previous header
 floorKey(dataRowIndex)
 Pro
 Minimal changes to business logic
 Con
 Relationships may be too complex
ID Type Data
1 Head …
1 Data …
1 Data …
2 Head …
2 Data …
3 Head …
3 Data …

51. 51
Apache Spark – Challenge 1: Independent Processing, Solution 3
Modify business logic
 Row relationship could be removed, if it’s
 Unintentional (a bug)
 For unnecessary optimization
 Data that could be retrieved otherwise
 Pro
 High chance for good performance
 Con
 High chance for new bugs

52. 52
Apache Spark – Challenge 2: I/O
 Input and output data must be shared
 Network storage
 How long does it take to copy 200 GB?
Transfer
Process
Transfer
Process
Transfer
Heavy
Process
Heavy
ProcessTransfer
Transfer Process

53. 53
Spark – Challenges – Challenge 3: Data Ordering
 Sequential batches rely on ordering
 Tricky to keep in Spark
 Safe operations: map, filter, zip
 Unsafe operations: join, group, sort
Process
Process
Process
Process
Process
Process
Shuffle
Process
Process
Process
Shuffle

54. 54
Spark Takeaways
 Good for
 Heavy processing
 Independent input data records
 One input, multiple outputs
 Unordered data
 Not so great for
 Little processing
 Dependencies between data records
 Merging multiple data sources

55. 55
5. Judgement Day

56. Ready for Migration
56

57. 57
Migration – Schedule
321
321Data
Saturday Sunday Monday

58. 58
Performance – Achieved!
vs.

59. 59
6. Into the Future

60. 60
Next Phase
1.0
Initial phase
2.0 In-house
development
3.0
Standardization
4.0
Data Optimized
Outsource based,
just started.
Vendor locked-in.
In-house
development,
differentiate with
lower costs and
faster delivery.
Standardized
system
architecture, both
for hardware and
software.
Overwhelming
differentiation,
with enabling
architecture for
customer centric
service.
Achieved Next
Current Standard
Architecture

61. THANK YOU