Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[KGC 2012] Online Game Server Architecture Case Study Performance and Security

게임 서버 구조 성능과 보안 중심으로

  • Be the first to comment

[KGC 2012] Online Game Server Architecture Case Study Performance and Security

  1. 1. Performance and Security 신승민 CTO ㈜윈디소프트 Online Game Server Architecture Case Study
  2. 2. Performance Security Server Architecture Case Study Publishing Platform
  3. 3. 論語 『 雍也 』 6-28 己欲立而立人 己欲達而達人
  4. 4. On game, what is my role as system engineer?
  5. 5. “TCP vs. UDP 속도는 뭐가 빠를까?” “ODBC vs. ADO 뭐가 좋을까?” “분산 시스템 어떻게 구축해야 하는가?” “HDD vs. SSD 뭘 선택해야 할까?” “데이터베이스 접속을 위하여 DBC를 따로 두는 것이 좋을까?” “개발자와 엔지니어 어떤 역할을 수행해야 할까?” “게임 서버도 가상화를 도입할 수 있을까?” “암호와 압축은 어떤 순서로 동작하는가?” “IP 주소 vs. DNS 기반 어떤 것이 효율적일까?” Questionnaire●
  6. 6. “Scale-out vs. Scale-up 선택은?” “DB 분산 솔루션 좋은 것 없을까?” “가상화는 어떤 이점이 있는 거지?” “Memcached 서버를 게임 서버에도 적용할 수 있을까?” “클라이언트 패치 효율적인 방안은?” “그럼 서버 패치 효율적인 방안은?” “서버에 안티바이러스 제품 설치해야 하는 건가?” “RAID 10 또는 RAID5 어느 것을 선택해야 하지?” “SSD에서 RAID 구성은?” Questionnaire●
  7. 7. Books – 2004年●
  8. 8. Lobby Gateway PvP World NAT Database DBC Validation Login GameDungeon Terms Chat Management Relay
  9. 9. Sample of Server Configuration● N모사의 C모 게임○ Rookie Level CPSvr CPSvr CPSvr RoomSvr GameDB AuthSvr Client Client UserDB PatchSvr NATSvr DBGW UDP Game Data DBGW
  10. 10. “Room Server: 게임 룰과 방 관리를 담당하는 서버” “CP Server: 병렬적으로 수가 증가할 수 있는 구조” Sample of Server Configuration● “CP Server: 연결 접속점 서버” Function of Servers○ “Patch Server: 사용자가 게임을 시작했을 때 현재 설치된 클라이언트 버전이 최근 버전과 동일한지 확인” “NAT Server: UDP Hole Punch를 담당하는 서버” “Auth Server: 로그인하는 작업 처리”
  11. 11. Sample of Server Configuration● “CP Server와 GameDB 통신 중간에 데이터베이스 미들웨어를 두어 데이터베이스 때문에 CP Server의 성능이 떨어지는 것을 최대한 방지” Function of Servers○ Rookie Level CPSvr CPSvr CPSvr RoomSvr GameDB AuthSvr Client Client UserDB PatchSvr NATSvr DBGW UDP Game Data DBGW
  12. 12. Sample of Server Configuration● How can a client connect among CP Servers?○ “사용자 접속 이전에 어느 CP Server로 접속할지를 관리해 주는 서버를 따로 만드는 것” “L4 스위치를 이용하는 방법” “클라이언트에서 CP Server 주소 테이블을 가지고 랜덤하게 접속을 시키는 방법” “가장 간단하지만 접속량을 고르게 분포하지 못함” “접속 위치 관리 서버도 병렬로 구성하는 등의 작업이 많이 필요하다” “대부분 웹 서비스를 서비스할 때 사용자를 분산시켜주기 위해서 사용하는 방법이지만 비싸다”
  13. 13. Rookie Level Sample of Server Configuration● Single point of failure○ CPSvr CPSvr CPSvr RoomSvr GameDB AuthSvr Client Client UserDB PatchSvr NATSvr DBGW UDP Game Data DBGW ? ? ? ? ? ? ?
  14. 14. Books – 2011~2012年●
  15. 15. High Availability Virtualization Scale-out vs. Scale-up Cloud Distribute system Open Source CompilerScalability Terms Synchronous vs. Non-synchronous system KVS
  16. 16. Books – 2001~2010年●
  17. 17. Cryptography Stream Cipher Kerckhoff Protocol Block CipherSSL CAPTCHA Key: Private, Public, Session, Symmetric Algorithm Security Hash Terms RSA
  18. 18. Online Game Server Platform Network Program DB Chat LobbyLogin Business Logic Cryptography The three key functions for game server●
  19. 19. Online Game Server Platform Game + Dungeon + NPC + PvP World + Channel + Lobby Authentication + Login Chat + Messenger + Mail + Friends Management + Duplicate login + OP Tools Logical Architecture● Module
  20. 20. Online game server Architecture
  21. 21. Build a new game server architecture● Client and Server○ Lobby Client ① Web Launcher ② Game Client
  22. 22. Build a new game server architecture● Authentication Servers○ Lobby1 Client Web ① What is the IP address or Hosts name of Authentication Server? ② Auth1 Auth1 UserDB
  23. 23. Build a new game server architecture● Patch Servers○ Lobby1 Client Web ① What is the IP address or Hosts name of Authentication Server? ② Auth1 Auth1 UserDB Patch1
  24. 24. Game Server Farm Build a new game server architecture● Lobby & Game Servers○ Lobby1 Client Web ① What is the IP address or Hosts name of Lobby Server? Lobby2 Lobby3 ③ Lobby1 Game1 Game2 Game3 ④ Game1
  25. 25. Game Server Farm Build a new game server architecture● Web Server Farm○ Lobby1 Client ① What is the IP address or Hosts name of Lobby Server? Lobby2 Lobby3 ③ Lobby1 Game1 Game2 Game3 ④ Game1 Web Server Farm Web1 Web2 Web3 L4 Virtualization
  26. 26. Build a new game server architecture● Authentication system○ Client ① ID & Password for authentication? Web Server Farm Web1 Web2 Web3 L4 Virtualization ② Authentication Cookie
  27. 27. Build a new game server architecture● Authentication system○ Client ① ID & Password for authentication? Web Server Farm Web1 Web2 Web3 L4 Virtualization ② Authentication Cookie ③ Authentication Cookie = ? Auth1 GameUserDB
  28. 28. Authentication Cookie● • 7aec312v1285345f31073df1e745caf6aeb0d3 cc0c9690b100b6c3011898a5b0d3fa22a98f52 cb6xf4c53e0344a1fe20c75670b01c310daab9 5a31b75f426a48722c7fbc16f8ec05e7988d36 8b94735b017048d77cffeb060cb20a602f8be3 e8f3ed6f8f43fffddc7bb7cdc6606df9df7a33b6 o42 javascript: document.cookie○
  29. 29. Game Server Farm Build a new game server architecture● Game Database Server○ Lobby1 Client WebLobby2 Lobby3 Game1 Game2 Game3 Auth1 UserDB Patch1 GameDB
  30. 30. K Online Architecture C/S MMORPG
  31. 31. ● Online game system○ Client Internet Game Server Game DB ① Client display & Business logic process ② Client side internet bandwidth ③ Business logic process ④ Database performance Bottleneck points
  32. 32. “화면에 보여주는 데이터를 어떻게 효과적으로 처리할지” ● “이미지 데이터를 어떻게 작게 만들지” Client display & Business logic process○ “Video RAM을 어떻게 효율적으로 올려둘지” “맵 사이즈 & 구조에 따라 어떻게 유저를 효과적으로 배치할지” Bottleneck points
  33. 33. ● Client Display○ Bottleneck points
  34. 34. ② 인스턴스법(인스턴스 던전) ● ① 공간 분할법(공간의 지리적 분할) Game Server & Database Performance○ ③ 패러렐 월드 방식 “게임의 세계를 지리적 구조에 근거하여 분할하여 다른 서버 프로세스, 혹은 서버 머신에 처리를 할당한다” “부하가 높고 유저가 집중하는 부분만에 전용 서버를 할당 한다. 그 부분은 반드시 던전일 필요는 없지만, 일반적으로 던전에서 사용되므로 인스턴스 던전이라고 불린다.” “보틀넥 발생이 쉬운 백엔드 DB 자체를 여러 개로 병렬화하여 보틀넥을 나눈다.” Bottleneck points resolve
  35. 35. ● One game server○  Game DB  Game Server Bottleneck points resolve
  36. 36. ● Space Partition Method○ Game DB  Game Server  Game Server  Game Server  Game Server Bottleneck points resolve
  37. 37. ● Space Copy Method○ Game DB  Game Server  Game Server  Game Server  Game Server “②랜덤으로 선택하게 한다” “①명시적으로 선택하게 한다” “③비어 있는 곳을 자동적으로 선택한다” Bottleneck points resolve
  38. 38. ● Instance Dungeon Method○  Game DB  Game Server ID#1 ID#2 ID#3 Bottleneck points resolve
  39. 39. ● Parallel World Method○ Game DB#1 Game Server Game DB#2 Game Server Game DB#3 Game Server Game DB#4 Game Server Bottleneck points resolve
  40. 40. ● Use in combination with Parallel World and Space Copy Method○ Game DB#1 Game Server Game Server Game Server Game Server Game DB#3 Game Server Game Server Game Server Game Server Game DB#2 Game Server Game Server Game Server Game Server Game DB#4 Game Server Game Server Game Server Game Server Bottleneck points resolve
  41. 41. ● All in One○ Game DB#1 Game Server Game Server Game Server Game Server 1 2 3 Game DB#3 Game Server Game Server Game Server Game Server 1 2 3 Game DB#2 Game Server Game Server Game Server Game Server 1 2 3 Game DB#4 Game Server Game Server Game Server Game Server 1 2 3 Bottleneck points resolve
  42. 42. ● Character Database○ Bottleneck points resolve Game DB#1 Server Server Server Server 1 2 3 Game DB#2 Server Server Server Server 1 2 3 Game DB#3 Server Server Server Server 1 2 3 Game DB#4 Server Server Server Server 1 2 3 Billing DB Character DB
  43. 43. Web: 10% < 90% Game: 90% > 10% “SSD: Solid State Drive” Write : Read○ “KVS: Key-Value Store” How to resolve○ “Database Table Compress” Compare between Web and Game DB●
  44. 44. Scale Out vs. Scale Up
  45. 45. Scale horizontally (scale out)● Database scalability: http://en.wikipedia.org/wiki/Scalability○ “One technique supported by most of the major database management system (DBMS) products is the partitioning of large tables, based on ranges of values in a key field. In this manner, the database can be scaled out across a cluster of separate database servers”
  46. 46. Scale vertically (scale up)● ○ “With the advent of 64-bit microprocessors, multi-core CPUs, and large SMP multiprocessors, DBMS vendors have been at the forefront of supporting multi-threaded implementations that substantially scale up transaction processing capacity” Database scalability: http://en.wikipedia.org/wiki/Scalability
  47. 47. “Storage Area Network” Scale up● “HDD vs. SSD” Database: Hardware spec○
  48. 48. “RAID Configuration” Scale up● “RPM(Revolutions Per Minute)” Hardware○ “BUS Interface”
  49. 49. “RPM” Scale up● High speed disk○ HDD Spindle [rpm] Average rotational latency [ms] 4,200 7.14 5,400 5.56 7,200 4.17 10,000 3.00 15,000 2.00 Game Database
  50. 50. Scale up● “RAID Configuration” http://en.wikipedia.org/wiki/RAID○
  51. 51. Level Minimum # of drives Read Benefit Write Benefit Image RAID 0 2 nX nX RAID 1 2 nX 1X RAID 4 3 (n-1)X (n-1)X RAID 5 3 (n-1)X (n-1)X
  52. 52. Nested (hybrid) RAID● • RAID 0+1: striped sets in a mirrored set (minimum four drives; even number of drives) provides fault tolerance and improved performance but increases complexity. – The key difference from RAID 1+0 is that RAID 0+1 creates a second striped set to mirror a primary striped set. The array continues to operate with one or more drives failed in the same mirror set, but if drives fail on both sides of the mirror the data on the RAID system is lost. • RAID 1+0: (a.k.a. RAID 10) mirrored sets in a striped set (minimum four drives; even number of drives) provides fault tolerance and improved performance but increases complexity. – The key difference from RAID 0+1 is that RAID 1+0 creates a striped set from a series of mirrored drives. The array can sustain multiple drive losses so long as no mirror loses all its drives. From Wikipedia○
  53. 53. RAID 10 versus RAID 5● • A common opinion (and one which serves to illustrate the dynamics of proper RAID deployment) is that RAID 10 is inherently better for relational databases than RAID 5, because RAID 5 requires the recalculation and redistribution of parity data on a per-write basis. • While this may have been a hurdle in past RAID 5 implementations, the task of parity recalculation and redistribution within modern storage area network (SAN) appliances is performed as a back-end process transparent to the host, not as an in-line process which competes with existing I/O. (i.e. the RAID controller handles this as a housekeeping task to be performed during a particular spindle's idle timeslices, so as not to disrupt any pending I/O from the host.) The "write penalty" inherent to RAID 5 has been effectively masked since the late 1990s by a combination of improved controller design, larger amounts of cache, and faster drives. The effect of a write penalty when using RAID 5 is mostly a concern when the workload cannot be de-staged efficiently from the SAN controller's write cache. From Wikipedia○
  54. 54. RAID 10 versus RAID 5● • The choice between RAID 10 and RAID 5 for the purpose of housing a relational database depends upon a number of factors (spindle availability, cost, business risk, etc.) but, from a performance standpoint, it depends mostly on the type of I/O expected for a particular database application. For databases that are expected to be exclusively or strongly read-biased, RAID 10 is often chosen because it offers a slight speed improvement over RAID 5 on sustained reads and sustained randomized writes. If a database is expected to be strongly write-biased, RAID 5 becomes the more attractive option, since RAID 5 does not suffer from the same write handicap inherent in RAID 10; all spindles in a RAID 5 can be utilized to write simultaneously, whereas only half the members of a RAID 10 can be used. However, for reasons similar to what has eliminated the "read penalty" in RAID 5, the 'write penalty' of RAID 10 has been largely masked by improvements in controller cache efficiency and drive throughput. From Wikipedia○
  55. 55. Scale up● http://en.wikipedia.org/wiki/SATA○ Name Raw bandwidth(Mbit/s) Transfer speed(MB/s) eSATA 3,000 300 SATA revision 3.0 6,000 600 SATA revision 2.0 3,000 300 SATA revision 1.0 1,500 150 USB 3.0 5,000 400 USB 2.0 480 60 Fibre Channel over optic fibre 10,520 1,000 Fibre Channel over copper cable 4,000 400 Thunderbolt 10,000 1,250 “Bus Interface”
  56. 56. Performance Charts HDD vs. SSD● HDD: Write Access Times: score (in ms)○ [SRC] http://www.tomshardware.com/charts/hard-drives-and-ssds,3.html
  57. 57. Performance Charts HDD vs. SSD● HDD: Read Access Times: score (in ms)○
  58. 58. Performance Charts HDD vs. SSD● SSD: Write Access Times: score (in ms)○ x80
  59. 59. Performance Charts HDD vs. SSD● SSD: Read Access Times: score (in ms)○
  60. 60. Performance Charts HDD vs. SSD● SSD: Sequential Write: score (in MB/s)○
  61. 61. Performance Charts HDD vs. SSD● SSD: Sequential Read: score (in MB/s)○
  62. 62. Angelbird Crest 6 Master Samsung MZ-7PC256N Samsung MCCOE64G5MPP IOMeter Web Server Benchmark 6537.10 6876.00 1241.85 IOMeter 4K Random Reads Benchmark 37298.00 58946.00 6166.33 IOMeter Database Benchmark 9883.00 8910.00 319.63 CrystalDiskMark Random Read, 4KB(QD =32) [MB/s] 212.00 319.00 25.22 CrystalDiskMark Random Write, 4KB(QD =32) [MB/s] 19.00 154.00 0.76 CrystalDiskMark Sequential Read, [MB/s] 184.00 403.00 100.54 CrystalDiskMark Sequential Write, [MB/s] 262.00 537.00 78.96 CrystalDiskMark Random Read, 4KB(QD =1) [MB/s] 33.00 26.00 24.27 CrystalDiskMark Random Write, 4KB(QD =1) [MB/s] 512.00 121.00 0.74 Performance Charts HDD vs. SSD● SSD: Compare with Angelbird and Samsung○ x5 x6 x30 x8 x25 x1.8 x3.3 x1.3 x691 x5 x9 x27 x12 x202 x4 x6 x1 x163
  63. 63. Performance Charts HDD vs. SSD● ORCA Disk Run Percent○ HDD SSD
  64. 64. RAID configuration for SSD● Papers○
  65. 65. Flash SSD상의 RAID 구성의 고찰● Papers○  Intel’s iOmeter를 이용하여 테스트
  66. 66. Flash SSD상의 RAID 구성의 고찰● Papers○
  67. 67. Flash SSD상의 RAID 구성의 고찰● Papers○
  68. 68. Flash SSD상의 RAID 구성의 고찰● Papers○
  69. 69. Flash SSD상의 RAID 구성의 고찰● Papers○
  70. 70. Scale out● “a-1, b-2, … , z-26” Distribute Game Database System by Partitioning○ RDBMS . . . Game DB(1) Game DB(26) Login Server Lobby ServerUser . . . . . . . . .
  71. 71. Scale out● “Hash(user id) = n” Distribute Game Database System by Partitioning○ Hash(user_id) = n RDBMS . . . Game DB(1) Game DB(26) Login Server Lobby ServerUser . . . . . . . . .
  72. 72. Game Server Farm Build a new game server architecture● View of high availability○ Lobby1 Client Web Lobby2 Lobby3 Game1 Game2 Game3 Patch1Auth Database Server Farm GameDB UserDB M M S S Replication
  73. 73. Scale Out● “KVS(Key-Value Store)” Database: Middleware○ User RDBMS Game DB Login Server Lobby Server . . . . . . . . . . . . KVS Server
  74. 74. Cache● Write-through vs. Writer-back○ “Write-back(Write-behind)” Write is done synchronously both to the cache and to the backing store. “Write-through” Writing is done only to the cache. A modified cache block is written back to the store, just before it is replaced.
  75. 75. Write-through vs. Write-back● • Write-back cache is more complex to implement, since it needs to track which of its locations have been written over, and mark them as dirty for later writing to the backing store. The data in these locations are written back to the backing store only when they are evicted from the cache, an effect referred to as a lazy write. For this reason, a read miss in a write-back cache (which requires a block to be replaced by another) will often require two memory accesses to service: one to write the replaced data from the cache back to the store, and then one to retrieve the needed datum. http://en.wikipedia.org/wiki/Cache_(computing)#Writing_policies○
  76. 76. Write-through vs. Write-back●
  77. 77. http://ihchoi.tistory.com/11○ Compress of Database Table● * 샘플 데이터 1000만 건 Page 압축 時 48.5% 감소, Row 압축 時 77.8% 감소
  78. 78. http://ihchoi.tistory.com/11○ Compress of Database Table● CPU 118% 조금 증가, Page 압축 時 35.6% 감소, Row 압축 時 32.8% 감소
  79. 79. http://ihchoi.tistory.com/11○ Compress of Database Table● CPU 235% 증가, Page 압축 時 249.8% 증가, Row 압축 時 150.8% 증가 * 100만 건 입력 시간 비교
  80. 80. Moor’s law Multi-core processor Memory SSD Virtualization Scale-out vs. Scale-up●
  81. 81. ② 5개 패러렐 월드로 분할하여 하나의 월드는 동시 접속 6,000을 허용, 합계 3만의 동시 접속을 허용한다 ① 과금 인증 시스템은 공통으로 한다 시스템 기본 구조도의 작성○ ③ 하나의 월드를 8개의 지역으로 분할한다(8코어) ④ 하나의 월드당 360개의 인스턴스를 준비한다(16코어) K Online Game System●
  82. 82. K Online Game System● Billing System K Online World2 W3 W4 W5 WEB 1 5 1 World1 2 6 2 3 7 4 8 3 6 0 Field Instance Game DB CU 6,000名
  83. 83. K Online Game System● 결제SV Backend DBMS CU 6,000名 Backup dbsv DBMS Backup logsv worldsv authsv Frontend Internet loginsv gmsv1 msgsv gmsv2 gmsv3 proxy proxy proxy proxy proxy cli cli cli 공간 분할법을 이용했을 경우의 프로세스 관계도○
  84. 84. Case Study● Game Client Web Board
  85. 85. W Publishing Platform● WAG: Authentication WPG: Payment WPIN: Personal Identification Number WOTP: One Time Password Game Security○ WPCBG: PCBang WIRG: Item Register Game Support○
  86. 86. W Publishing Platform● WAG CashDB WPG WPCBG PC Bang DB Game Server XML-RPC JSON AVRO Channel Game Client AuthDB JDBC JDBC x
  87. 87. XML-RPC is a remote procedure call (RPC) protocol which uses XML to encode its calls and HTTP as a transport mechanism JSON-RPC is a remote procedure call protocol encoded in JSON. It is a very simple protocol (and very similar to XML-RPC), defining only a handful of data types and commands. http://en.wikipedia.org/wiki/XML-RPC○ http://en.wikipedia.org/wiki/JSON-RPC○ Remote procedure call (RPC) protocol● Avro is a remote procedure call and serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format. http://en.wikipedia.org/wiki/Thrift_(protocol)○
  88. 88. W Personal Identification Number●
  89. 89. W One Time Password●
  90. 90. How to hack Diablo 3, I’ve OTP?
  91. 91. 2channel Authentication system● Case by Dream Security○
  92. 92. 2channel Authentication system● Compare○ 2channel + 2factor 1channel + 2 factor 2channel + 2factor ARS
  93. 93. SHA-3● http://csrc.nist.gov/groups/ST/hash/sha-3/Round3/index.html○
  94. 94. SHA-3● • The NIST hash function competition is an open competition held by the US National Institute of Standards and Technology for a new SHA-3 function to replace the older SHA-1 and SHA-2, which was formally announced in the Federal Register on November 2, 2007. • NIST has selected five SHA-3 candidate algorithms to advance to the third (and final) round:  BLAKE  Grøstl (Knudsen et al.)  JH  Keccak (Keccak team, Daemen et al.)  Skein (Schneier et al.) http://csrc.nist.gov/groups/ST/hash/sha-3/Round3/index.html○
  95. 95. SHA-3● • NIST noted some factors that figured into its selection as it announced the finalists:  Performance: "A couple of algorithms were wounded or eliminated by very large [hardware gate] area requirement – it seemed that the area they required precluded their use in too much of the potential application space."  Security: "We preferred to be conservative about security, and in some cases did not select algorithms with exceptional performance, largely because something about them made us 'nervous,' even though we knew of no clear attack against the full algorithm."  Analysis: "NIST eliminated several algorithms because of the extent of their second-round tweaks or because of a relative lack of reported cryptanalysis – either tended to create the suspicion that the design might not yet be fully tested and mature."  Diversity: The finalists included hashes based on different constructions, including the HAIFA and sponge hash constructions, and hashes with different sources of nonlinearity, including S-boxes and the interaction between addition and XOR.
  96. 96. SHA-3● • NIST noted some factors that figured into its selection as it announced the finalists:  Performance: "A couple of algorithms were wounded or eliminated by very large [hardware gate] area requirement – it seemed that the area they required precluded their use in too much of the potential application space."  Security: "We preferred to be conservative about security, and in some cases did not select algorithms with exceptional performance, largely because something about them made us 'nervous,' even though we knew of no clear attack against the full algorithm."  Analysis: "NIST eliminated several algorithms because of the extent of their second-round tweaks or because of a relative lack of reported cryptanalysis – either tended to create the suspicion that the design might not yet be fully tested and mature."  Diversity: The finalists included hashes based on different constructions, including the HAIFA and sponge hash constructions, and hashes with different sources of nonlinearity, including S-boxes and the interaction between addition and XOR.
  97. 97. SHA-3 WINNER● • NIST announced Keccak as the winner of the SHA-3 Cryptographic Hash Algorithm Competition and the new SHA-3 hash algorithm in a press release issued on October 2, 2012. Keccak was designed by a team of cryptographers from Belgium and Italy, they are: – Guido Bertoni (Italy) of STMicroelectronics, – Joan Daemen (Belgium) of STMicroelectronics, – Michaël Peeters (Belgium) of NXP Semiconductors, and – Gilles Van Assche (Belgium) of STMicroelectronics.
  98. 98. Books●
  99. 99. 8F Seung Kwang Bldg., 143-8 Samsung-Dong, Gangnam-Gu, Seoul, 135-877, Korea 신 승 민 / Seungmin Shin TEL: +82-2-2050-4307 E-MAIL: joo @ windysoft.net 감사합니다! CONTACT ME

×