This document describes a batch system that moves data from MongoDB to MySQL databases. It includes: 1. An overview of the batch system architecture with MongoDB as the source and MySQL as the target database. 2. Diagrams showing the sequence of operations for a single and multiple batch apps to transfer the data. 3. Details about schema designs for the target MySQL tables to store specific data types from MongoDB like ObjectIDs. 4. Code snippets from a sample student batch app that reads from MongoDB and writes to corresponding MySQL tables.

1. The Solution for Technical
Test
Claire Kim
1

2. Contents
1. Test Scenario and Plan
2. Batch System between MongoDB and MySQL
3. The sequence diagram of batch system
4. The sequence diagram of number of Batch Apps
5. RDB Batch App Structure
6. Issues of Converting Data to MySQL
7. RDB Schema - Organisation
8. RDB Schema - Role
9. RDB Schema - Country
10. RDB Schema - Product
11. RDB Schema - Teacher
12. RDB Schema - Student
13. RDB Schema - Player
14. The call Sequence of RDB Batch App
15. Some Code of RDB Batch App - Student
16. Test Environment
2

3. 1. Test Scenario and Plan
3

4. 1. Test Scenario and Plan
4

5. 2. Batch System between MongoDB and MySQL
Master
Slave Slave -
Hidden
Slave
Special Tag for mirror to make dump at downtime.
MySQL
MongoDB
MongoDB Dump Server
RDB Batch Apps
RDB Batch App
1. Read Data from Dump server
2. Write the data on MySQL
3. Remove the data from Dump
server
5
Dump Server has only modified data list from
hidden Slave. E.g. lastModifiedTime in a day

6. 3. The sequence diagram of batch system
● One Batch App can be operated.
● Can set up how many number of
data will be read from RDB Batch
App in App.Conf file.
● The data on MongoDB is not
removed if update/insert of it is
fail on MySQL.
● Batch App read again the data
and try to write on MySQL.
6

7. 4. The sequence diagram of number of Batch Apps
● Multiple Batch App can be
operated if the estimation time of
completion to write on MySQL.
● It can be estimated by the size of
dump or the list of data.
● Recommend to read one
data from MongoDB.
○ Set up the
number of read
data = 1
7

8. Connection Pooling on MongoDB
5. RDB Batch App Structure
Connections of dump server
Connections of dump server
Connection Pooling on MySQL
Connections of MySQL
Connections of MySQL
Data
Read and
Transfer
On Thread
(Single)
8

9. 6. Issues of Converting Data to MySQL
1. Convert ObjectID(bson) on MongoDB to VARCHAR(25) on MySQL
2. Integer primary keys on MySQL to be used efficiently.
Example MongoDB MySQL
Student oid ObjectID(12) VARCHAR(25)
Organisation oid ObjectID(12) VARCHAR(25)
MySQL
Field Data Type
StudentID BIGINT(20), Primary Key
StudentOID VARCHAR(25), Unique
● The issues can be solved by the purpose to use MySQL Database.
9

10. 7. RDB Schema - Organisation
Organisation has the number of their classes.
ClassID can be unique with organisation ID by the
requirements.1 : N
10

11. 8. RDB Schema - Role
Role Table
List of RoleID
1. Teacher
2. Student
3. Teacher & Tester
4. Student & Tester
5. Developer
11

12. 9. RDB Schema - Country
Country Table
1. country Code is unique.
2. Language Code is unique.
countryID can be created as unique ID or the
combination of city code and country code. It
depends on how it can be used in other tools.
12

13. 10. RDB Schema - Product
Product Table
productID is unique and can have multiple branchIDs
1 : N
13

14. 11. RDB Schema - Teacher
2
3
5
4
6
1. roleID in role Table
2. countryID in Country Table
3. classID in Organisation Table.
4. productID in Product table
5. branchID in ProductBranch
table
1 1 : N
1 : 1 1 : 1
14

15. 12. RDB Schema - Student
1
2
4
5
1. studentID is created in the insert with
auto_increasement.
studentOID is ObjectID of MongoDB.
2. roleID in role Table
3. studientID of StudentProfile is used in
StudentLoginConfig table as primary
and foreign keys.
4. productID in Product table
5. branchID in ProductBranch table3
1 : 1
● Add fields - CreatedTimeByBatch, LastUpdatedTimeByBatch in
StudentProfile and StudentLoginConfig Tables will be useful.
15

16. 13. RDB Schema - Player
1
2 4
5
1. playerID is created by MySQL as integer type.
Student can have the number players if possible.
2. playerOID is ObjectID in Student Document of
MongoDB.
3. studentID is uniqueID of StudentProfile Table.
4. uID is unique PlayerEquipmentID like item unique ID.
Player can have the number of equipments like hairs.
5. playerID is the primary key of Player Table.
3
1 : N
16

17. 14. The call Sequence of RDB Batch App
11. If update failed, the data doesn’t b
removed. The data will be read by 3.
4. The number of find data can be set up in App
Store
procedure
17

18. 15. Some Code of RDB Batch App - Student
18

19. 15. Some Code of RDB Batch App - Student
19

20. 15. Some Code of RDB Batch App - Student
20

21. 15. Some Code of RDB Batch App - Student
21

22. 15. Some Code of
RDB Batch App -
Student
22

23. 16. Test Environment
● OS : Debian 8
● MongoDB : v3.6.3
● MySQL : 10.2.14-MariaDB
● Development Tool : Visual Studio Community 2017
23