The document outlines the transition of Rakuten Card Co., Ltd. from outdated core systems to modern architectures using Java EE 7 and Apache Spark, detailing the challenges faced, strategies implemented, and improvements achieved. It describes the development of new core systems that allow scaling and better security while eliminating vendor lock-in through the use of open-source technologies like Payara. Furthermore, it emphasizes the standardization and optimization of the system architecture for enhanced business operations and customer-centric services.

1. Oct 28, 2017
Hirofumi Iwasaki
Arshal Ameen
Rakuten Card Co., Ltd.

2. 2
 Hirofumi Iwasaki @HirofumiIwasaki
 Group Manager
 Technology Strategy Group, System Strategy Department,
Rakuten Card Co., Ltd.
 Career
 Planning, designing & implementation of huge enterprise systems for financial,
manufacturing and public systems with Java EE in Japan over 18 years.
 Opus, Lectures, etc.
 Conferences: OOW 2017, 2014, JavaOne 2017, 2015, 2014, Java Day Tokyo
2014-2015, Rakuten Tech Conference 2013-2016, etc.

3. 3
Part 1
Java EE 7 with Apache Spark for the
World’s Largest Credit Card Core
Systems
 Old Core Systems and Issues to
Solve
 Strategies and Designs of new Core
Systems
 Development and Results
Part 2
New Core Systems and Beyond
– Java EE 8 and Payara

4. 4
 Top-level credit card
company in Japan
 Core of Rakuten eco
systems.
 3rd position of total
transaction volume in 2016.
Growing rapidly.

5. 5
Core Systems
Web Systems
External Systems
Intra Systems

6. 6
Mainframe
 Old architecture – over 20 years
 High cost structure
 Capacity and performance
limitation – no scale out
 Low maintainability with piled
programs and old architecture
database "NDB"
 Risk against vendor locked-in
 Limitation of the security for the
significant data

7. 7
Business
Operations
Development

8. 8
Old New
 Cannot scale-out  Apply scale-out enabled
architecture, with Oracle RAC
and clustered WebLogic server.
 Low connectivity to other
systems
 Apply Java EE and latest
protocol.
 Less security management on
data
 Apply Oracle database
security options.
 No latest auto testing
environment
 Introduce latest auto testing
environment.

9. 9
Old New
 No local development  Apply Java EE and Oracle DB
for local dev.
 Hard to understand because
of its old architecture
 Apply latest Java EE for its
basement.
 Poor version control systems  Introduce git server and issue
track systems.
 No development community  Apply Java EE and join open
community.

10. 10
Old New
 Poor automated operations  Introduce Jenkins and
automations.
 Manual error monitoring  Include Zabbix monitoring to
cover the new core system.
 Difficult to pin-point cause of
error
 Use standard Java tools: stack
traces, Flight Recorder, etc.
 Tons of unused codes  Apply automated source code
analyzing tool.

11. 11
Phase of the improvement – 3.0
1.0
Initial phase
2.0 In-house
development
3.0
Standardization
4.0
Data Optimized
Outsource based,
just started.
Vendor locked-in.
In-house
development,
differentiate with
lower costs and
faster delivery.
Standardized
system
architecture, both
for hardware and
software.
Overwhelming
differentiation,
with enabling
architecture for
customer centric
service.
Achieved Next
Current Standard
Architecture

12. 12
2013 2017
Web systems
Expand
Java EE 6
Core
Systems
Java EE 7

13. 13
Oracle Exalogic
+ Exadata + ZFS Servers
13
Mainframe
Old New
Core
Systems

14. 14
14
Oracle Exalogic
+ Exadata + ZFS Servers
Oracle Cloud Machine
(On premise private cloud)
For temporarily
request spiking
Low-Cost
Temp
Resource
New
Core
Systems

15. 15
 Financial de-facto standard
 Java EE compliant.
 Matured, from 1997.
 Financial de-facto standard
 ISO/IEC 9075 SQL compliant
 Matured, from 1983.
COBOL
Network
DB
App Server
Database
Old New
WebLogic Server
Oracle Database

16. 16
Hardware, OS, app
server independent,
vendor free.
Mainframe,
Japanese COBOL,
vendor locked-in
Old New
Widfly
Payara
WebLogic
hp-ux
AIXSolaris
Linux
Windows
macOS
WebSphere
Java EE 7

17. 17
Japanese COBOL
Source code
Source code
Custom made
source code
converter
 Convert from Japanese
COBOL to Java EE
 Keep original core
business logic
Java EE

18. 18
From Web Systems,
For New Logic
COBOL
From Old System,
converted to Java
 Ease of migration, resource re-use
 Introduce power of Java EE
 Introduce converter from YPS to Java
“Dual Source Architecture”
Japanese
COBOL
 Japanese source code
 Almost abandoned
 No books, no community
Old New
Java EE

19. 19
BIG-IP
Real-time Servers
(WebLogic)
Batch Servers
(Spark & Java)
Façade
Rich clients Façade
Façade
Intranet
External
Intra
Exadata
Mail
Form
BIG-IP
Façade
BIG-IP
External
customers
Scheduler
CoreBusinessLogicAPIs
Operation
terminal
Web
browser
Old New
Java EE
Java

20. 20
New Database
ISAM
VSAM
NDB
Java
Business Logics
Japanese COBOL
business logics
Common Module
Data Accessor Common Module
Database Accessor
Migrate
Web
Database
Old New

21. 21
New Database
- Record Key
(Unique)
- Record Key
(Not Unique)
A_RDB_TABLE
----------------------------
- PRIMARY KEY
- OTHER COLUMN
Add unique
index.
Add index
only.
Old New
ISAM/VSAM

22. 22
3. Run tests
on staging environment
2. Execute auto testing
on several times
1. Register auto test scenarios
 Automatic testing
using latest IBM
Rational test software.
 Regression test
enabled when
something changed.
 Reduce error
possibilities on
production release.
Testing
Server

23. 23
Time
Sequential
Parallel

24. 24
Cluster Node
Cluster Node
Cluster Node
Cluster Node
Cluster Node
Cluster Node
Bootstrap
SharedMemory
Scheduler

25. 25
1. Making business logic parallel
 Independent processing
2. I/O
 Data transferred over network
3. Data ordering
 Shuffles

26. 26
 Good for
 Heavy processing
 Independent input data records
 One input, multiple outputs
 Unordered data
 Not so great for
 Little processing
 Dependencies between data records
 Merging multiple data sources

27. 27
vs.

28. 28
Next Phase
1.0
Initial phase
2.0 In-house
development
3.0
Standardization
4.0
Data Optimized
Outsource based,
just started.
Vendor locked-in.
In-house
development,
differentiate with
lower costs and
faster delivery.
Standardized
system
architecture, both
for hardware and
software.
Overwhelming
differentiation,
with enabling
architecture for
customer centric
service.
Achieved Next
Current Standard
Architecture

29. 29
 Arshal Ameen
 Web System Development
Team Manager
 Core Engineer and trainer
 Carrier
 Java EE expert
 App developer (iOS, Android and Win).
 Opus, Lectures, etc.
 Conferences: JavaOne 2015, 2014, Oracle OpenWorld 2014,
Rakuten Tech Conf 2016-2013, JJUG Tokyo(2014)

30. 30
Part 1
Java EE 7 with Apache Spark for the
World’s Largest Credit Card Core
Systems
 Old Core Systems and Issues to
Solve
 Strategies and Designs of new Core
Systems
 Development and Results
Part 2
New Core Systems and Beyond
– Moving on to Payara

31. 31

32. 32
Drop in replacement for glassfish(currently orphaned)
Enhancements to GlassFish, bug fixes, patches to GlassFish server and
dependent libraries
Quarterly community releases
Open source @ www.github.com/payara (RakutenCard is already a code
contributor)
Synched with Glassfish SVN

33. 33
GlassFish
PAYARA
EXPERTISE

34. 34
Feature GlassFish Payara
Releases/Upgrades No plan Quarterly
Patches None after Glassfish4.0 Monthly (Licensed)
Quarterly (Community)
Security Fixes Infrequent Instant (Licensed)
ASAP (Community)
Support None ✓
Caching Shoal Shoal/Jcache/Payara scales
Clustering None Hazelcast
Admin command recorder None ✓
SQL logging/Monitoring
logging
None ✓
Healthcheck service None ✓

35. 35
Feature Weblogic Payara
Releases/Upgrades Delayed Quarterly
Security Patches 2016 Security bugs
2 resolved
18 pending
2016 Security bugs
All resolved
Support Cost $$$$ 18750$/6 Processor
Caching OracleWebCache (buggy) Shoal/Jcache/Payara scales
Clustering Vendor specific Hazelcast
Server start time ~10 mins ~ 12 sec
Open source No ✓
Critical Hotfixes (~1hr
support)
No ✓

36. 36

37. 37
Need to setup a clustered domain with auto session replication ????
asadmin create-domain --nopassword --portbase 10000 example-domain
asadmin start-domain example-domain
asadmin -p 10048 create-cluster sample-cluster
asadmin -p 10048 set-hazelcast-configuration --enabled true --dynamic true –
multicastport 5000 --target sample-cluster
asadmin -p 10048 set sample-cluster-config.availability-service.web-container
availability.availability-enabled=true
asadmin -p 10048 set sample-cluster-config.availability-service.web-container
availability.persistence-type=hazelcast
 Save this as script and create ∞ domains in a jiffy(technically 33.33ps)
 Schedule automatic backups
 Restore from a backup or create new from backup

38. 38

39. 39

40. 40

41. 41

42. 42
 healthcheck-cpool Connection Pool
 healthcheck-cpu CPU usage
 healthcheck-gc Garbage collection
 healthcheck-heap Heap memory
 healthcheck-threads Threads
 healthcheck-machinemem Machine storage
asadmin>
Step1: Enable
healthcheck-configure --enabled=true --dynamic=true
Step 2: Configure frequency
healthcheck-configure-service --serviceName=healthcheck-cpu --time=5
--unit=MINUTES
Step3: Configure threshold
healthcheck-configure-service-threshold --thresholdCritical=95 --
thresholdWarning=75 --thresholdGood=60

43. 43
JDBC connection pools
HTTP connection pool
GC activity
Stuck threads

44. 44

45. 45
Hazelcast
 Hazelcast’s JCache cached In-memory
 WAN replication across multiple DCs
 Provides resilience and high availability
 Disaster recovery – multiple copies of sessions across
cluster for HA
 High Density Memory Store scales
 Standardized

46. 46
• Standard Java API for Caching
JSR107
• API and CDI binding
• Supported by Many Cache Providers
• Hazelcast, EhCache, Coherence, Infinispan,
Apache Ignite
• Built in to Payara Server
Uses Hazelcast

47. 47
• In‐Memory DataGrid(IMDG)
• Clustering and highly scalable
data distribution solution forJava
• Distributed Data Structures for Java
• Distributed Hashtable(DHT) and more

48. 48
• Open source(ApacheLicense)
• Superlight,simple,no‐dependency
• Distributed/partioned implementation of map,queue,set,list,lock
and executor service
• Transactional(JCAsupport)
• Dynamic clustering,backup,fail‐over
• Partion/Share data across cluster
• Send/receive messages
• Scaling and load balancing
• Process in parallel on many JVM

49. 49

50. 50