Your SlideShare is downloading. ×
0
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

[RakutenTechConf2013] [A-4] The approach of Event in Japan Ichiba

695

Published on

Rakuten Technology Conference 2013 …

Rakuten Technology Conference 2013
"The approach of Big Event in Japan Ichiba"
Yusuke Kobayashi, Osamu Iwasaki, Makito Hashiyama (Rakuten)

Published in: Technology, Design
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
695
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
18
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • We already implemented this logic.But VIP re-allocation process is on going now.At now, 5% of CPU resources are already shift to new VIP group.
  • Transcript

    • 1. The approach of Big Event In Japan Ichiba Vol.01 Oct/26/2013 Yusuke Kobayashi Group Manager Mall Group, Japan Ichiba Section Rakuten Ichiba Development Department, Rakuten, Inc. http://www.rakuten.co.jp/
    • 2. Index Big Sale in Ichiba 2
    • 3. Introduce Me Yusuke Kobayashi Group Manager Japan Ichiba Section Japan Mall Group Rakuten Ichiba Development Department Japan Ichiba Section MALL RMS IBS IBE • Joined in 2005. • Fist career in Rakuten was Infoseek. • Transferred to Ichiba from 2009. 3
    • 4. Index 1.Scale of Big Sales - Huge Traffic Scale, Amazing Sales. 2.History of success - Share how we could improve our services. 3.Case study - Show trouble case and explain the countermeasure. -- Checkout System -- Infrastructure(Cloud Environment/Network) 4
    • 5. Shopping Marathon Shop around points 5
    • 6. Super Sale Half Price Items, Point, Topic Items 6
    • 7. Victory Sale 77%Off, Half Price, 1001 yen items 7
    • 8. 1.Scale of Big Sales - Huge Traffic Scale, Amazing Sales. 2.History of success - Share how we could improve our services. 3.Case study - Show trouble case and explain the countermeasure. -- Checkout System -- Infrastructure(Cloud Environment/Network) 8
    • 9. 1.Scale of Big Sale 15 Billion Sales per day 9
    • 10. 1.Scale of Big Sale 5% traffic of entire Japan!! Victory Sale 10
    • 11. 1.Scale of Big Sale Comparison of order numbers between big sale and usual. Order Number Sale Usual 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 Time(1hour) 11
    • 12. 1.Scale of Big Sales - Huge Traffic Scale, Amazing Sales. 2.History of success - Share how we could improve our services. 3.Case study - Show trouble case and explain the countermeasure. -- Checkout System -- Infrastructure(Cloud Environment/Network) 12
    • 13. Monitoring by 100 DU members 13
    • 14. Our Energy! 14
    • 15. 2. History 2012/03/04 00:00 – 2012/03/04 23:59 2012/06/03 00:00 – 2012/03/06 01:59 2012/12/02 00:00 – 2012/12/04 01:59 2013/03/03 00:00 – 2013/03/05 01:59 2013/06/02 00:00 – 2013/06/05 01:59 2013/09/01 00:00 – 2013/09/04 01:59 2013/09/27 00:00 – 2013/09/30 01:59 15
    • 16. 2012/03 – Super Sale TV Commercial Train AD Special Items 24h Limited Half Price 16
    • 17. 2012/03 – Super Sale Event Page Top Page Entry Search Item Page Bookmark Review Purchase History Search Engine AD ID Point Coupon Checkout 17
    • 18. 2012/03 – Super Sale 18
    • 19. 2012/03 – Super Sale 19
    • 20. 2012/03 – Super Sale  Almost all services Delayed.  Huge traffic.  Application high load.  Over band frequency.  DB high load.  NFS high load. Mass Media power was so huge. 20
    • 21. 2012/03 – Super Sale  Just only Restart and Reboot.  Change Apache Configuration.  Restart Apache.  Reboot physical servers.  Contents Delete  Access Control by Creative Web Design team. 21
    • 22. 2012/03 – Super Sale  Countermeasure       Enhance Web/Apps/NW/DB Servers. Band width limitation Tuning Middleware configuration. Decrease traffic. Contents Control. Web Front Speed UP. 22
    • 23. 2012/06 – Super Sale 23
    • 24. 2012/06 – Super Sale Top Page Event Page Entry Search Item Page Bookmark Review Purchase History Search Engine AD ID Point Coupon Checkout 24
    • 25. 2012/06 – Super Sale  ID went down and Checkout delayed. We expanded the period due to big troubles. 25
    • 26. ID Service 0:00AM Sun 6/3 2:00AM Mon 6/4 60,000 [0:41] [22:37 - 0:42] Login Success DB Connection Error terminated just after stopping batch program for Fraud Access Management running every 10 min. Users became login smoothly. Ichiba stopped using ID service. Purchases were only processed by non-members. 50,000 40,000 [Sun#login 20120603 Sale (2nd) this time 6/3] Super [Sun#login 20120527b 5/27] Ordinary Sunday #login 20120304b [Sun, 3/4] Super Sale (1st) last time 30,000 20,000 10,000 0 6/3 0:00 6/3 2:00 6/3 4:00 6/3 6:00 6/3 8:00 6/3 10:00 6/3 12:00 6/3 14:00 6/3 16:00 6/3 18:00 6/3 20:00 6/3 22:00 6/4 0:00 6/4 2:00 [23:15 - 23:40] Fig.1: # of Login Successes [0:00-1:09] Just after the launch of Super Sales, DB Connection Errors occurred because of the users' massive accesses. Some of the users experienced connection errors. Errors automatically solved with users’ access decrease. ID service stopped because of server reboot and server down. 400000 [20:20 - 0:34] 350000 Serious DB Connection Errors occurred because of the users' massive accesses. Critical user login failures by reboots of the ID services, limitation of Login, etc. 300000 250000 200000 150000 100000 ID3 2/3 ID3 1/4 50000 Fig.2: # of DB Connection Errors 2012/6/4 2:30 2012/6/4 2:00 2012/6/4 1:30 2012/6/4 1:00 2012/6/4 0:30 2012/6/4 0:00 2012/6/3 23:30 2012/6/3 23:00 2012/6/3 22:30 2012/6/3 22:00 2012/6/3 21:30 2012/6/3 21:00 2012/6/3 20:30 2012/6/3 20:00 2012/6/3 19:30 2012/6/3 19:00 2012/6/3 18:30 2012/6/3 18:00 2012/6/3 17:30 2012/6/3 17:00 2012/6/3 16:30 2012/6/3 16:00 2012/6/3 15:30 2012/6/3 15:00 2012/6/3 14:30 2012/6/3 14:00 2012/6/3 13:30 2012/6/3 13:00 2012/6/3 12:30 2012/6/3 12:00 2012/6/3 11:30 2012/6/3 11:00 2012/6/3 9:30 2012/6/3 10:30 2012/6/3 9:00 2012/6/3 10:00 2012/6/3 8:30 2012/6/3 8:00 2012/6/3 7:30 2012/6/3 7:00 2012/6/3 6:30 2012/6/3 6:00 2012/6/3 5:30 2012/6/3 4:30 2012/6/3 4:00 2012/6/3 3:30 2012/6/3 3:00 2012/6/3 2:30 2012/6/3 2:00 2012/6/3 1:30 2012/6/3 1:00 2012/6/3 0:30 2012/6/3 0:00 0 2012/6/3 5:00 6/2 22:00 26
    • 27. ID Service  ID Service had serious DB connection errors during the following time period. (1) 0:00 - 1:09 (Sun, 6/3-2012) Just after the launch of Super Sales, DB Connection Errors occurred because of the users' massive accesses. Some of the users experienced connection errors. # of DB Connection Errors = 160,497 (in 10 min) (ref. # of Login Successes = 2,167,126) (2) 20:20(Sun, 6/3-2012) - 0:34 (Mon, 6/4) Serious DB Connection Errors occurred because of the users' massive accesses. Critical user login failures by reboots of the ID services, limitation of Login, etc. 22:37-0:42, Ichiba stopped using ID service. Purchases were only processed by non-members. # of DB Connection Errors = 4,063,667 (in 4 hours 15 min) Impacted to entire Rakuten Group. 27
    • 28. Checkout Enhance the instances APP APP APP Web APP Web APP Web APP APP API APP APP APP Change the threshold APP APP APP APP APP APP APP APP APP APP APP APP Web APP APP Web APP APP Web APP APP Change the configuration of thread APP APP Web APP Web APP Web APP APP APP Enhance the instances 28
    • 29. Have no time!!!! 29
    • 30. 2012/06 – Super Sale  Countermeasure for next Super Sale.  Migration of DB servers.(ID)  Enhancement Applications Servers. -> Transfer to Cloud environment not using Physical servers.  Load Test on Production environment. -> We did on staging environment, but it was not enough. Different between • User Numbers • Item Numbers staging and • Transaction production • Server Spec environment. 30
    • 31. 2012/12 – Super Sale 31
    • 32. 2012/12 – Super Sale Top Page Event Page Entry Search Bookmark Item Page Review Purchase History Search Engine AD ID Point Coupon Cart 32
    • 33. 2012/12 – Super Sale  Search, Item Page and Checkout were down. The first peak time -> Down This was the most high traffic in this year. 33
    • 34. Search Application load was high APP APP APP APP APP APP APP APP Web APP Web APP APP Web APP APP APP Search Engine APP APP APP APP APP APP APP APP APP APP APP APP APP APP Enhance LB was high  The beginning of the first peak time, search applications was high load because the huge traffic came from event contents. -> We enhanced 65ins by using Cloud environment within 4 hours.  The LB which we enhanced was high load. 34
    • 35. Item Page 12/02 9:00 pm to 24:00 Disk Util was 100% 35
    • 36. Item Page The connection delayed between App and NFS. 36
    • 37. Item Page We switched to Akamai during peak time.  Cache 25min.(by Mikitani-san suggestion.)  Inventory data was not updated in real time.  The countermeasure of emergency. 37
    • 38. Item Page 38
    • 39. Checkout APP High Load APP APP Web APP Web APP Web APP APP API APP APP APP APP APP APP APP APP APP APP APP APP APP APP APP Web APP APP Web APP APP Web APP APP APP APP Web APP Web APP Web APP APP APP 39
    • 40. 2012/12 – Super Sale  Countermeasures for next    Continuously doing the load test of checkout system. Need to decrease NFS call numbers in Item Page system. Transfer to Cloud environment gradually. 40
    • 41. Item Page  AppServer  Decreasing Unnecessary File Call  Memcached Server  Cache more kinds of files. - Shop data, Layout data.  Re-cache anytime whether file is update or not when cache is expired 41
    • 42. Before Item Page After 30% Down 42
    • 43. Checkout  Load Test about 50 times in midnight… APP Jmeter Cache Web WEB server APP APP server DataBase APP API α APP API β APP API γ APP API δ 43
    • 44. Checkout We always have so many load tests for checkout systems before Super Sale. Explain later…by Hashiyama. 44
    • 45. 2013/03 – Super Sale 45
    • 46. 2013/03 – Super Sale Top Page Event Page Entry Search Bookmark Item Page Review Purchase History Search Engine Cart AD ID Point Coupon STEP BasketAPI 46
    • 47. No Trouble!! Success!! 47
    • 48. 2013/06 & 2013/09 – Super Sale No Big Trouble! 48
    • 49. From this year… 49
    • 50. Victory Sale Just only 3 weeks for this event preparation.. 50
    • 51. Victory Sale 1st Spike.  From Yahoo! 2nd Spike.  Event Started. 3rd Spike.  New Paper AD 51
    • 52. Victory Sale Top Page Event Page Entry Search Bookmark Item Page Review Purchase History Search Engine Cart AD ID Point Coupon STEP BasketAPI 52
    • 53. Top & Search Top Page Search  The traffic was higher more than we expected. Around 6 or 7 times!!!!  The countermeasure for this was just enhancement. 53
    • 54. 1.Scale of Big Sales - Huge Traffic Scale, Amazing Sales. 2.History of success - Share how we could improve our services. 3.Case study - Show trouble case and explain the countermeasure. -- Checkout System -- Infrastructure(Cloud Environment/Network) 54
    • 55. About Me  Name  Makito Hashiyama(@capyogu)  Role  Team manager of APIs for Rakuten Ichiba  Recent activity  GlassFish Community Feedback @ JavaOne 2013  Contact  makito.hashiyama@mail.rakuten.com 55
    • 56. Overview of Rakuten Ichiba Checkout  Architecture  Behaves like a service bus based on SOA  Calls more than 15 external APIs and mashes up them, then provides External APIs  Scale  More than 100 application servers SOAP/ REST Client side Checkout API KVS 56
    • 57. Overview of Rakuten Ichiba Checkout  Provides over 50 services to client side  Get/Set shopper information  Get/Set merchant information Merchant API  Validation Item API  Update inventory Cart API  Register order data into DB  etc.. SOAP/ REST Client side Checkout API 57
    • 58. Overview of Rakuten Ichiba Checkout  Stateful API  Manage session information instead of client side  Creates unique key and manage it with KVS  Client side only have to call API with the key key SOAP/ REST Client side value Key1 Checkout API KVS Session1 Key2 Session2 58
    • 59. Overview of Rakuten Ichiba Checkout  Rakuten Super Sale  Biggest online sales in Japan  It causes a huge amount of traffic  Performance bottleneck  External APIs called by our API were slow down  We needed to improve the system at the peak time Slow down delay Delay SOAP/ REST External API Checkout API 59
    • 60. How to execute load test  Environment  On production at midnight  Execute over 50 times with JMeter  Test case  100,000 dummy shoppers  1,000,000 dummy items / 6,500 dummy merchant  Reproduce sale’s load as much as possible delay JMeter SOAP/ REST External API Checkout API 60
    • 61. How to execute load test  As a result, bottleneck moved to APP server APP Jmeter Cache Web WEB server APP APP server DataBase APP API α APP API β APP API γ APP API δ 61
    • 62. Improvement to handle a huge traffic Worker Thread Request Worker Thread Task Queue Worker Thread Worker Thread Client side Worker Thread delay delay delay External APIs delay delay Checkout API CPU load was high 62
    • 63. Improvement to handle a huge traffic Request Worker Thread Task Queue Worker Thread External APIs Worker Thread Client side Checkout API (1)According to vmstat, ‘run queue’ was very high (2)Decrease worker threads to keep ‘run queue’ low (3)As a result, latency increased but throughput was improved 63
    • 64. Improvement to handle a huge traffic  As a result…  Checkout API could process over 12,000 transactions / minute  We also achieved 30,000 TPM in load test (Just yesterday we did!!!) 64
    • 65. Overview of Rakuten Ichiba Checkout In the future  Set SLA for each external API  Resolve performance issues  Synchronous vs. Asynchronous  Upgrade library / middleware / JDK  Deep copy(copy constructor vs. serialize) 65
    • 66. Self introduction Name : Osamu Iwasaki Role : Network / Cloud Eng & Mgr Twitter @osamuiwasaki Skype osamu.iwasaki Vice Group Manager Server Platform Group / Network administration Group Global Infrastructure Development Department And Committee member of JANOG(JApan Network Operators’ Group) 66
    • 67. Our traffics history (Gbps) 160 140 120 100 80 60 40 20 0 Peak traffics at Victory Sales, over 140Gbps which was about over 5% of Japan Internet traffics 67
    • 68. Our traffics history (Gbps) 160 140 Victory Sale 120 100 80 Super Sales 60 40 20 0 Peak traffics at Victory Sales, over 140Gbps which was about over 5% of Japan Internet traffics 68
    • 69. Network traffic trend from 2012/Jan(SS traffic focus) (Gbps) 160 140 120 100 80 60 40 20 0 SuperSale CDN 2012 June 2012 Dec 2012 Mar 2013 June 2013 Sep 2013 Oct(VS) 60G 78.9G 69.1G 75.8G 73.7G 127.6G RakutenDC 12.7G 14.2G 12.8G 12.5G 11.7G 12.9G Total 72.7G 93.1G 81.9G 88.3G 85.4G 140.5G 69
    • 70. PC/FeaturePhone/Smartphone/Table share by Sales Mobile traffics increase rapidly!! Almost 50% 70
    • 71. Our private cloud history About 1years ago, we starts from 300VMs. But now, around 10000VMs running for Rakuten Ichiba services. Compared last year is over 30 times !!! 71
    • 72. Victory Sale Just only 3 weeks for this event preparation.. 72
    • 73. But !! 73
    • 74. Our Load Balancers are downed…… 74
    • 75. What happened at peak time Peak Time LoadBalancer-A CPU utilization LoadBalancer-B CPU utilization Due to heavy traffics at VictorySale start time, CPU load of LoadBalancer rapidly growth…… 75
    • 76. After the result of re-allocation operation LoadBalancer-A CPU utilization LoadBalancer-B CPU utilization After the VIP re-allocation, we could separate heavy traffics to other LoadBalancer 76
    • 77. Our counter action for next Victory sale Internet Internet VIP Group A VIP Group A VIP Group B VIP Group B VIP Group B ActiveSLB (Target CPU under 15%) StandbySLB (Target CPU under 15%) ActiveSLB (Target CPU under 30%) StandbySLB (CPU 0%) Regular time 3times peak capable BigSale time 6times peak capable 3times is not enough for us, 6times we need for the Super/Victory sales. 77
    • 78. Next Victory sale ready? 78
    • 79. Next Victory sale ready? Yes, we are ready !!! 79
    • 80. Wrap UP 80
    • 81. Wrap Up • Traffic : 5% of entire Japan. • Sales : Over 15B yen/day • Continuously Tuning/Improvement • Cloud environment 81
    • 82. Global Expansion - Super Sale 82
    • 83. In future Worldwide Rakuten Super Sale 83
    • 84. And… 84
    • 85. 85
    • 86. 86
    • 87. Thank you for listening. Yusuke Kobayashi Makito Hashiyama Osamu Iwasaki @okoba23 @capyogu @osamuiwasaki 87

    ×