DOCOMO CLOUD
Mick Etoh, Ph.D.
November 13, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not b...
Who am I?

2
Friday, November 15, 13
Who am I?
•

Mick Etoh, SVP, NTT DOCOMO

2
Friday, November 15, 13
Who am I?
•

Mick Etoh, SVP, NTT DOCOMO

•

Role: R&D Strategy

2
Friday, November 15, 13
Who am I?
•

Mick Etoh, SVP, NTT DOCOMO

•

Role: R&D Strategy

•

Expertise:

2
Friday, November 15, 13
Who am I?
•

Mick Etoh, SVP, NTT DOCOMO

•

Role: R&D Strategy

•

Expertise:
Pattern Recognition & Machine Learning,
Sign...
Who am I?
•

Mick Etoh, SVP, NTT DOCOMO

•

Role: R&D Strategy

•

Expertise:
Pattern Recognition & Machine Learning,
Sign...
Who am I?
•

Mick Etoh, SVP, NTT DOCOMO

•

Role: R&D Strategy

•

Expertise:
Pattern Recognition & Machine Learning,
Sign...
Who is DOCOMO?

3
Friday, November 15, 13
Who is DOCOMO?
•

$61 million subscribers in Japan

3
Friday, November 15, 13
Who is DOCOMO?
•

$61 million subscribers in Japan

•

$45 billion revenue (2013)

3
Friday, November 15, 13
Who is DOCOMO?
•

$61 million subscribers in Japan

•

$45 billion revenue (2013)
Next to China Mobile, Verizon, and
AT&T
...
Who is DOCOMO?
•

$61 million subscribers in Japan

•

$45 billion revenue (2013)
Next to China Mobile, Verizon, and
AT&T
...
Revenue Structure (FY2012)
Feature Phone Era
Data Stage
Voice Stage
Service
Cloud

Packet
Network
Voice
Network

Device

4...
Revenue Structure (FY2012)
Feature Phone Era
Data Stage
Voice Stage

Smart Phone

Service
Cloud

Service $5.5 Billion

Pac...
Wireless Carrier + Service Provider

5
Friday, November 15, 13
DOCOMO CLOUD Services

6
Friday, November 15, 13
DOCOMO CLOUD Services
docomo Phonebook

docomo Mail

Photo Collection

databox

dmarket

Schedule and Memo

ToruCa

Hanash...
Service Delivery Platform RENEWED

7
Friday, November 15, 13
Service Delivery Platform RENEWED

DOCOMO CLOUD
7
Friday, November 15, 13
Enterprises using Public Cloud in Japan

8
Friday, November 15, 13
Enterprises using Public Cloud in Japan

2.5% (2011)

Yano Research

8
Friday, November 15, 13
Enterprises using Public Cloud in Japan

7.3% (2012)

Yano Research

8
Friday, November 15, 13
DOCOMO as a Big AWS User

9
Friday, November 15, 13
DOCOMO as a Big AWS User
Four-Digit
EC2 Instances

9
Friday, November 15, 13
A Telco Story About Launching
Voice-Command Personal Agent
Service with AWS Cloud
10
Friday, November 15, 13
Siri

11
Friday, November 15, 13
“Shabette-Concier” Voice agent service

12
Friday, November 15, 13
“Shabette-Concier” Voice agent service
Concier

=

=

How may I help you?

Shabette
Voice

Concierge

•
•
•
•

Launched Ma...
13

Friday, November 15, 13
13

Friday, November 15, 13
14

Friday, November 15, 13
14

Friday, November 15, 13
15

Friday, November 15, 13
15

Friday, November 15, 13
An Entrance to Services

16
Friday, November 15, 13
An Entrance to Services

37

22

20

search types

applications

functions

Train transfer / Last train alarm / Weather al...
Monetization

“d-shopping” : real marketplace

Just say, wanna buy!
17

Friday, November 15, 13
Monetization
How may I help you?

I want to buy
water!

“d-shopping” : real marketplace

OK !

Just say, wanna buy!
17

Fr...
18
Friday, November 15, 13
“Status Quo”
Shrine

18
Friday, November 15, 13
“99.9999%”
Temple

“Status Quo”
Shrine

18
Friday, November 15, 13
“99.9999%”
Temple

“Status Quo”
Shrine

Office of
Inspirations

18
Friday, November 15, 13
“99.9999%”
Temple
Wannabe
Skunkworks

“Status Quo”
Shrine

Office of
Inspirations

18
Friday, November 15, 13
19
Friday, November 15, 13
In Status Quo Shrine,
so many superstitions....
19
Friday, November 15, 13
Is this ‘Cloud’?
In Status Quo Shrine,
so many superstitions....
19
Friday, November 15, 13
In 99.9999% Temple,

20
Friday, November 15, 13
In 99.9999% Temple,
Belief: High Availability Forever

20
Friday, November 15, 13
In 99.9999% Temple,
Belief: High Availability Forever

Downtime per Year
31.5 seconds
20
Friday, November 15, 13
Three Major Concerns to Public Cloud
Security
Performance
Switching Cost
21
Friday, November 15, 13
At Office of Inspirations

22
Friday, November 15, 13
At Office of Inspirations

Make a Magic Lamp
right now
with Genie (2009)

22
Friday, November 15, 13
In Wannabe Skunkworks (2010)

23
Friday, November 15, 13
In Wannabe Skunkworks (2010)
Maybe, it’s
high time to make
a voice agent.
But, as a stealth product.

23
Friday, November ...
In Wannabe Skunkworks (2010)
Maybe, it’s
high time to make
a voice agent.
But, as a stealth product.
Yep, we have lot of e...
In Wannabe Skunkworks (2010)

Just out of
curiosity,
I’d like to use Public
Cloud.

Maybe, it’s
high time to make
a voice ...
In Wannabe Skunkworks (2010)

Just out of
curiosity,
I’d like to use Public
Cloud.

Maybe, it’s
high time to make
a voice ...
Distributed Speech Recognition (2006)

First Implementation of ETSI AURORA Project (2000-2003)
16 Khz Sampling MFCC featur...
Distributed Speech Recognition (2006)
Mobile Terminal

Dictionary
Server

Feature
Extraction

Encode

Decode

Rec.
Engine
...
In Wannabe Skunkworks
Data Mining Team
(2009)

25
Friday, November 15, 13
In Wannabe Skunkworks
Data Mining Team
(2009)

Web 2.0 Believers
25
Friday, November 15, 13
The paradigm shift from ‘Search’

26
Friday, November 15, 13
The paradigm shift from ‘Search’
Microsoft Bing

Google Voice Search
26
Friday, November 15, 13
A Small Team in Wannabe Skunkworks (2010-2012)
Product owner
Development
promoter
Core engineer
Agile
Development
Core eng...
A Small Team in Wannabe Skunkworks (2010-2012)
Product owner
Development
promoter
Core engineer
Agile
Development
Core eng...
Basic Architecture 2010

28
Friday, November 15, 13
Basic Architecture 2010

(inspired by Microsoft Bing, and Google Voice Search)

28
Friday, November 15, 13
Basic Architecture 2010

(inspired by Microsoft Bing, and Google Voice Search)
text

Text to speech

contents

Voice

text...
History of Voice Agent
5/2011
Stealth Product

Data Center
29
Friday, November 15, 13

(Hybrid with other Cloud)
History of Voice Agent
5/2011
Stealth Product

3/2012
Version 1

Mar.∼
Data Center
Friday, November 15, 13

Public
Cloud

...
History of Voice Agent
5/2011
Stealth Product

3/2012
Version 1

Mar.∼
Data Center
Friday, November 15, 13

Public
Cloud

...
History of Voice Agent
5/2011
Stealth Product

3/2012
Version 1

Mar.∼
Data Center
Friday, November 15, 13

Public
Cloud

...
Voice Recognition
I want...

Speech Recognition

Text

Speech
Acoustic
Model

Machine Learning

Speech
Data

Lexicon

Lang...
Voice Recognition
I want...

Speech Recognition

Text

Speech
Acoustic
Model

Machine Learning

Speech
Data

Lexicon

Lang...
Task Recognition and Service Flow
Restaurant

Lexicon

Task Corpus

Dictionary
I want...

Tokenizer

Text
Abstractor

Cont...
Task Recognition and Service Flow
Restaurant

Lexicon

Task Corpus

Dictionary
I want...

Tokenizer

Text
Abstractor

Cont...
Version 1 Implementation (Mar. 2012)
Started with two local cloud
providers in Japan, and soon faced
two difficulties:

32...
Version 1 Implementation (Mar. 2012)
Started with two local cloud
providers in Japan, and soon faced
two difficulties:

LB...
Version 1 Implementation (Mar. 2012)
Started with two local cloud
providers in Japan, and soon faced
two difficulties:

LB...
Version 1 Implementation (Mar. 2012)
Started with two local cloud
providers in Japan, and soon faced
two difficulties:

LB...
Scrambling
•Mar 2012
0.5M accesses/day
•Apr 2012
1.5M accesses/day
•Jul 2012
2.5M accesses/day
33
Friday, November 15, 13
Scrambling
•Mar 2012
0.5M accesses/day
•Apr 2012
1.5M accesses/day
•Jul 2012
2.5M accesses/day
33
Friday, November 15, 13
Moved to AWS (June, 2012)

34
Friday, November 15, 13
Moved to AWS (June, 2012)

Scale!

34
Friday, November 15, 13
Moved to AWS (June, 2012)
Technology
Platform
Leader

AWS Partners
Innovative
Ecosystem

Scale!

Global User
Footprint

34...
System Architecture (June 2012)
SmartPhone

VPC

ELB
Management
Server

Log Server for VR

(across multiple zones)

Same a...
Killer Design Pattern: Multi-Data Center
Voice Recognition Part

Voice

VPC

Route 53

LB

VR servers

Availability Zone #...
DOCOMO Skunkworks: Cloud Natives
• For another service,
adopted 21 out of 48
design patterns.
• Among them, typically
used...
Other Crucial Design Patterns

38
Friday, November 15, 13
Other Crucial Design Patterns
•Scheduled Scale Out

38
Friday, November 15, 13
Other Crucial Design Patterns
•Scheduled Scale Out

38
Friday, November 15, 13
Other Crucial Design Patterns
•Scheduled Scale Out
•Clone Server

38
Friday, November 15, 13
Traffic Pattern
Normal traffic

0:00

1:00

2:00

3:00

4:00

5:00

6:00

7:00

8:00

9:00

10:00

11:00

12:00

13:00

14...
Traffic Pattern
Normal traffic

0:00

1:00

2:00

3:00

4:00

5:00

6:00

7:00

8:00

9:00

10:00

11:00

12:00

13:00

14...
Bursty traffic for which Auto-Scale fails
TV commercial causes a spike traffic.

40
Friday, November 15, 13
Bursty traffic for which Auto-Scale fails
TV commercial causes a spike traffic.
30x traffic
# of servers

0

1

2

3

4

5...
Bursty traffic for which Auto-Scale fails
TV commercial causes a spike traffic.
10min

30x traffic
# of servers

20:38

0
...
Unexpected Traffic
Scheduled
–TV Commercial
–Introduction by TV program
fortunately pre-scheduled!
Suddenly happened
–Eart...
Prepare for Launching Servers
Refresh AMI every day
–VR model has been
changed everyday
Launch AMI for bursty
traffic
–10 ...
More intelligence

24hours

43
Friday, November 15, 13
More intelligence

24hours

Optimized server deployment using
Elastic Map Reduce (EMR) for pattern analysis
43
Friday, Nov...
Moved twice

• Apr 2012
–Start to plan for moving-out to AWS
• Jun 2012
–Moved to AWS Northern California
• Tokyo did not ...
October, 2012

“Status Quo”
DOCOMO
“99.9999%”
Shrine
Skunkworks
Temple
Office of
Inspirations
(not ‘Wannabe’ anymore)
45
F...
October, 2012

Kid, your game
is over.

“Status Quo”
DOCOMO
“99.9999%”
Shrine
Skunkworks
Temple
Office of
Inspirations
(no...
October, 2012

Use our
stable system.

Kid, your game
is over.

“Status Quo”
DOCOMO
“99.9999%”
Shrine
Skunkworks
Temple
Of...
October, 2012

Use our
stable system.

Kid, your game
is over.

Move on
quickly!

“Status Quo”
DOCOMO
“99.9999%”
Shrine
Sk...
October, 2012
Catch me
if you can
SCALE!

Use our
stable system.

Kid, your game
is over.

Move on
quickly!

“Status Quo”
...
In DOCOMO Skunkworks (2013)

46
Friday, November 15, 13
In DOCOMO Skunkworks (2013)

They are SLOW and
pricy. Keep Restless
improvement and our leanstartup culture.

46
Friday, N...
In DOCOMO Skunkworks (2013)

They are SLOW and
pricy. Keep Restless
improvement and our leanstartup culture.
Yep, let’s
co...
In DOCOMO Skunkworks (2013)

Hey, leader,
how about multi-lingual
translation services?

They are SLOW and
pricy. Keep Res...
In DOCOMO Skunkworks (2013)

Hey, leader,
how about multi-lingual
translation services?

They are SLOW and
pricy. Keep Res...
Now, we have …

17-Nov-16 27-Nov-16 7-Dec-16 17-Dec-16 27-Dec-16 6-Jan-17 16-Jan-17 26-Jan-17 5-Feb-17 15-Feb-17 25-Feb-17...
Other Tips
• Instance Management
✓Simple DB as a key-value
store

• Log management

✓Enormous logs from servers
✓Direct wr...
Thanks to the people who gave great
lessons to ‘DOCOMO Skunkworks.’

49
Friday, November 15, 13
Thanks to the people who gave great
lessons to ‘DOCOMO Skunkworks.’
in changing our development style to

49
Friday, Novem...
Thanks to the people who gave great
lessons to ‘DOCOMO Skunkworks.’
in changing our development style to
“Deploy first, th...
Thanks to the people who gave great
lessons to ‘DOCOMO Skunkworks.’
in changing our development style to
“Deploy first, th...
Thanks to the people who gave great
lessons to ‘DOCOMO Skunkworks.’
in changing our development style to
“Deploy first, th...
Thanks to the people who gave great
lessons to ‘DOCOMO Skunkworks.’
in changing our development style to
“Deploy first, th...
Thanks to the people who gave great
lessons to ‘DOCOMO Skunkworks.’
in changing our development style to
“Deploy first, th...
Please give us your feedback on this
presentation

MBL202
As a thank you, we will select prize
winners daily for completed...
Upcoming SlideShare
Loading in …5
×

Telco: Voice-Command Personal Agent Service with AWS Cloud (MBL202) | AWS re:Invent 2013

2,280
-1

Published on

"In March 2012, Japan’s leading mobile operator, NTT DOCOMO, introduced Shabette Concier, an advanced voice-activated personal agent service that enables customers to intuitively and directly operate services and smartphone features with voice commands. Millions of DOCOMO's subscribers are now using this service.
This session explains Shabette Concier's distributed speech recognition architecture, and dialogue-understanding system design, with machine learning technologies and large-scale database systems. Learn why DOCOMO chose the AWS cloud and how DOCOMO engineers overcame all the difficulties from CEO-imposed time constraints, unexpectedly rapid service growth, usage spikes driven by marketing campaigns, and internal resistance to the use of cloud services. The session concludes with lessons learned from a telco's large-scale service development of a mobile app with the AWS cloud."

Published in: Technology, Business
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,280
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
82
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Telco: Voice-Command Personal Agent Service with AWS Cloud (MBL202) | AWS re:Invent 2013

  1. 1. DOCOMO CLOUD Mick Etoh, Ph.D. November 13, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. Friday, November 15, 13 1
  2. 2. Who am I? 2 Friday, November 15, 13
  3. 3. Who am I? • Mick Etoh, SVP, NTT DOCOMO 2 Friday, November 15, 13
  4. 4. Who am I? • Mick Etoh, SVP, NTT DOCOMO • Role: R&D Strategy 2 Friday, November 15, 13
  5. 5. Who am I? • Mick Etoh, SVP, NTT DOCOMO • Role: R&D Strategy • Expertise: 2 Friday, November 15, 13
  6. 6. Who am I? • Mick Etoh, SVP, NTT DOCOMO • Role: R&D Strategy • Expertise: Pattern Recognition & Machine Learning, Signal Processing & Multimedia, Mobile Networks 2 Friday, November 15, 13
  7. 7. Who am I? • Mick Etoh, SVP, NTT DOCOMO • Role: R&D Strategy • Expertise: Pattern Recognition & Machine Learning, Signal Processing & Multimedia, Mobile Networks Fun: 2 Friday, November 15, 13
  8. 8. Who am I? • Mick Etoh, SVP, NTT DOCOMO • Role: R&D Strategy • Expertise: Pattern Recognition & Machine Learning, Signal Processing & Multimedia, Mobile Networks Fun: Road Bike, Fly Fishing, and Wine Lover 2 Friday, November 15, 13
  9. 9. Who is DOCOMO? 3 Friday, November 15, 13
  10. 10. Who is DOCOMO? • $61 million subscribers in Japan 3 Friday, November 15, 13
  11. 11. Who is DOCOMO? • $61 million subscribers in Japan • $45 billion revenue (2013) 3 Friday, November 15, 13
  12. 12. Who is DOCOMO? • $61 million subscribers in Japan • $45 billion revenue (2013) Next to China Mobile, Verizon, and AT&T 3 Friday, November 15, 13
  13. 13. Who is DOCOMO? • $61 million subscribers in Japan • $45 billion revenue (2013) Next to China Mobile, Verizon, and AT&T • “i-mode” innovator (1999) 3 Friday, November 15, 13
  14. 14. Revenue Structure (FY2012) Feature Phone Era Data Stage Voice Stage Service Cloud Packet Network Voice Network Device 4 Friday, November 15, 13
  15. 15. Revenue Structure (FY2012) Feature Phone Era Data Stage Voice Stage Smart Phone Service Cloud Service $5.5 Billion Packet Network Data: $19 Billion Voice Network Voice: $13 Billion Device 4 Friday, November 15, 13
  16. 16. Wireless Carrier + Service Provider 5 Friday, November 15, 13
  17. 17. DOCOMO CLOUD Services 6 Friday, November 15, 13
  18. 18. DOCOMO CLOUD Services docomo Phonebook docomo Mail Photo Collection databox dmarket Schedule and Memo ToruCa Hanashite Hon'yaku Utsushite Hon'yaku dmenu Mail-Honyaku-Concier Shabette-Concier docomo DriveNet i-concier Sugotoku Containt 6 Friday, November 15, 13
  19. 19. Service Delivery Platform RENEWED 7 Friday, November 15, 13
  20. 20. Service Delivery Platform RENEWED DOCOMO CLOUD 7 Friday, November 15, 13
  21. 21. Enterprises using Public Cloud in Japan 8 Friday, November 15, 13
  22. 22. Enterprises using Public Cloud in Japan 2.5% (2011) Yano Research 8 Friday, November 15, 13
  23. 23. Enterprises using Public Cloud in Japan 7.3% (2012) Yano Research 8 Friday, November 15, 13
  24. 24. DOCOMO as a Big AWS User 9 Friday, November 15, 13
  25. 25. DOCOMO as a Big AWS User Four-Digit EC2 Instances 9 Friday, November 15, 13
  26. 26. A Telco Story About Launching Voice-Command Personal Agent Service with AWS Cloud 10 Friday, November 15, 13
  27. 27. Siri 11 Friday, November 15, 13
  28. 28. “Shabette-Concier” Voice agent service 12 Friday, November 15, 13
  29. 29. “Shabette-Concier” Voice agent service Concier = = How may I help you? Shabette Voice Concierge • • • • Launched Mar. 1, 2012 Over 40 services in it Including chatting 10 million users 12 Friday, November 15, 13
  30. 30. 13 Friday, November 15, 13
  31. 31. 13 Friday, November 15, 13
  32. 32. 14 Friday, November 15, 13
  33. 33. 14 Friday, November 15, 13
  34. 34. 15 Friday, November 15, 13
  35. 35. 15 Friday, November 15, 13
  36. 36. An Entrance to Services 16 Friday, November 15, 13
  37. 37. An Entrance to Services 37 22 20 search types applications functions Train transfer / Last train alarm / Weather alarm / Traffic information / Receipt search / Map search / Weather search / Image search / QA search / Movie search / Music search / News search / Twitter search / Game search / App search / Book/Comic search / Area guide / Browser search / Artist search / TV program search / Dictionary / Horoscope / Result search for soccer game / Result search for baseball game / Result search for golf tournament / Shopping / Calculate calorie of food / Sightseeing search / Call taxi / Knowledge of life / Q&A for DOCOMO / Phone call / Launch camera / Timer / Alarm / Scedular / Memo / Music player / Email / i-Concirge / i-Channel / DOCOMO Map / Disaster kit / Ir communication / Post Twitter / Photo collection / Launch Galary / Translation / Cheat sheet / User support / Flight mode / Day and time / Terminal information / Display setting / Keyboard and input / Storage setting / Language setting / Location service setting / Application listup / Security setting / Sound setting / User dictionary / Wi-Fi setting / Wall paper setting / Battery setting / Communication service setting 16 Friday, November 15, 13
  38. 38. Monetization “d-shopping” : real marketplace Just say, wanna buy! 17 Friday, November 15, 13
  39. 39. Monetization How may I help you? I want to buy water! “d-shopping” : real marketplace OK ! Just say, wanna buy! 17 Friday, November 15, 13
  40. 40. 18 Friday, November 15, 13
  41. 41. “Status Quo” Shrine 18 Friday, November 15, 13
  42. 42. “99.9999%” Temple “Status Quo” Shrine 18 Friday, November 15, 13
  43. 43. “99.9999%” Temple “Status Quo” Shrine Office of Inspirations 18 Friday, November 15, 13
  44. 44. “99.9999%” Temple Wannabe Skunkworks “Status Quo” Shrine Office of Inspirations 18 Friday, November 15, 13
  45. 45. 19 Friday, November 15, 13
  46. 46. In Status Quo Shrine, so many superstitions.... 19 Friday, November 15, 13
  47. 47. Is this ‘Cloud’? In Status Quo Shrine, so many superstitions.... 19 Friday, November 15, 13
  48. 48. In 99.9999% Temple, 20 Friday, November 15, 13
  49. 49. In 99.9999% Temple, Belief: High Availability Forever 20 Friday, November 15, 13
  50. 50. In 99.9999% Temple, Belief: High Availability Forever Downtime per Year 31.5 seconds 20 Friday, November 15, 13
  51. 51. Three Major Concerns to Public Cloud Security Performance Switching Cost 21 Friday, November 15, 13
  52. 52. At Office of Inspirations 22 Friday, November 15, 13
  53. 53. At Office of Inspirations Make a Magic Lamp right now with Genie (2009) 22 Friday, November 15, 13
  54. 54. In Wannabe Skunkworks (2010) 23 Friday, November 15, 13
  55. 55. In Wannabe Skunkworks (2010) Maybe, it’s high time to make a voice agent. But, as a stealth product. 23 Friday, November 15, 13
  56. 56. In Wannabe Skunkworks (2010) Maybe, it’s high time to make a voice agent. But, as a stealth product. Yep, we have lot of expertise & data for speech recognition. 23 Friday, November 15, 13
  57. 57. In Wannabe Skunkworks (2010) Just out of curiosity, I’d like to use Public Cloud. Maybe, it’s high time to make a voice agent. But, as a stealth product. Yep, we have lot of expertise & data for speech recognition. 23 Friday, November 15, 13
  58. 58. In Wannabe Skunkworks (2010) Just out of curiosity, I’d like to use Public Cloud. Maybe, it’s high time to make a voice agent. But, as a stealth product. Yep, we have lot of expertise & data for speech recognition. 23 Friday, November 15, 13
  59. 59. Distributed Speech Recognition (2006) First Implementation of ETSI AURORA Project (2000-2003) 16 Khz Sampling MFCC feature vectors --> 5.6 Kbps 24 Friday, November 15, 13
  60. 60. Distributed Speech Recognition (2006) Mobile Terminal Dictionary Server Feature Extraction Encode Decode Rec. Engine Result First Implementation of ETSI AURORA Project (2000-2003) 16 Khz Sampling MFCC feature vectors --> 5.6 Kbps 24 Friday, November 15, 13
  61. 61. In Wannabe Skunkworks Data Mining Team (2009) 25 Friday, November 15, 13
  62. 62. In Wannabe Skunkworks Data Mining Team (2009) Web 2.0 Believers 25 Friday, November 15, 13
  63. 63. The paradigm shift from ‘Search’ 26 Friday, November 15, 13
  64. 64. The paradigm shift from ‘Search’ Microsoft Bing Google Voice Search 26 Friday, November 15, 13
  65. 65. A Small Team in Wannabe Skunkworks (2010-2012) Product owner Development promoter Core engineer Agile Development Core engineer Development promoter 27 Friday, November 15, 13
  66. 66. A Small Team in Wannabe Skunkworks (2010-2012) Product owner Development promoter Core engineer Agile Development Core engineer Development promoter 27 Friday, November 15, 13
  67. 67. Basic Architecture 2010 28 Friday, November 15, 13
  68. 68. Basic Architecture 2010 (inspired by Microsoft Bing, and Google Voice Search) 28 Friday, November 15, 13
  69. 69. Basic Architecture 2010 (inspired by Microsoft Bing, and Google Voice Search) text Text to speech contents Voice text text contents Voice Recognition Logging Friday, November 15, 13 Task Recognition Logging Service Providers’ DB 28
  70. 70. History of Voice Agent 5/2011 Stealth Product Data Center 29 Friday, November 15, 13 (Hybrid with other Cloud)
  71. 71. History of Voice Agent 5/2011 Stealth Product 3/2012 Version 1 Mar.∼ Data Center Friday, November 15, 13 Public Cloud 29 (Hybrid with other Cloud)
  72. 72. History of Voice Agent 5/2011 Stealth Product 3/2012 Version 1 Mar.∼ Data Center Friday, November 15, 13 Public Cloud June AWS NC-reg 29 (Hybrid with other Cloud)
  73. 73. History of Voice Agent 5/2011 Stealth Product 3/2012 Version 1 Mar.∼ Data Center Friday, November 15, 13 Public Cloud 11/2012 Version 2 June AWS NC-reg Sept. AWS Tokyo-reg. (Hybrid with other Cloud) 29
  74. 74. Voice Recognition I want... Speech Recognition Text Speech Acoustic Model Machine Learning Speech Data Lexicon Language Model Machine Learning Language Data 30 Friday, November 15, 13
  75. 75. Voice Recognition I want... Speech Recognition Text Speech Acoustic Model Machine Learning Speech Data Lexicon Language Model Machine Learning Language Data BELIEF: More data usually beats better algorithms. 30 Friday, November 15, 13
  76. 76. Task Recognition and Service Flow Restaurant Lexicon Task Corpus Dictionary I want... Tokenizer Text Abstractor Contents Feature extractor Abstraction Dictionary MC-SVM Classifier Query ext. Find a good Italian restaurant in Palo Alto 31 Friday, November 15, 13
  77. 77. Task Recognition and Service Flow Restaurant Lexicon Task Corpus Dictionary I want... Tokenizer Text Abstractor Contents Feature extractor Abstraction Dictionary MC-SVM Classifier Service Launcher Search Engine A Query ext. Search Engine B Search Engine C Find a good Italian restaurant in Palo Alto PRINCIPLE: Machine learning enhances the service quality. A better quality service acquires more data. Friday, November 15, 13 31
  78. 78. Version 1 Implementation (Mar. 2012) Started with two local cloud providers in Japan, and soon faced two difficulties: 32 Friday, November 15, 13
  79. 79. Version 1 Implementation (Mar. 2012) Started with two local cloud providers in Japan, and soon faced two difficulties: LB A B 32 Friday, November 15, 13
  80. 80. Version 1 Implementation (Mar. 2012) Started with two local cloud providers in Japan, and soon faced two difficulties: LB A B • Scalability in server count 32 Friday, November 15, 13
  81. 81. Version 1 Implementation (Mar. 2012) Started with two local cloud providers in Japan, and soon faced two difficulties: LB A B • Scalability in server count • Inflexible monthly payment 32 Friday, November 15, 13
  82. 82. Scrambling •Mar 2012 0.5M accesses/day •Apr 2012 1.5M accesses/day •Jul 2012 2.5M accesses/day 33 Friday, November 15, 13
  83. 83. Scrambling •Mar 2012 0.5M accesses/day •Apr 2012 1.5M accesses/day •Jul 2012 2.5M accesses/day 33 Friday, November 15, 13
  84. 84. Moved to AWS (June, 2012) 34 Friday, November 15, 13
  85. 85. Moved to AWS (June, 2012) Scale! 34 Friday, November 15, 13
  86. 86. Moved to AWS (June, 2012) Technology Platform Leader AWS Partners Innovative Ecosystem Scale! Global User Footprint 34 Friday, November 15, 13
  87. 87. System Architecture (June 2012) SmartPhone VPC ELB Management Server Log Server for VR (across multiple zones) Same as AZ #1 TR Servers VR Servers Voice Recognizer(VR) Availability Zone #1 (across multiple zones) LB ELB ELB (across multiple zones) Task Recognizer(TR) Availability Zone #2 Tokenizer Access Log Servers Log management system Availability Zone #3 35 Friday, November 15, 13
  88. 88. Killer Design Pattern: Multi-Data Center Voice Recognition Part Voice VPC Route 53 LB VR servers Availability Zone #1 HAProxy LB m2.4xlarge x ~300 !! VR servers Availability Zone #2 36 Friday, November 15, 13
  89. 89. DOCOMO Skunkworks: Cloud Natives • For another service, adopted 21 out of 48 design patterns. • Among them, typically used ‘Queuing  Chain’   and  ‘Scale  Out’  Pa4erns. 37 Friday, November 15, 13
  90. 90. Other Crucial Design Patterns 38 Friday, November 15, 13
  91. 91. Other Crucial Design Patterns •Scheduled Scale Out 38 Friday, November 15, 13
  92. 92. Other Crucial Design Patterns •Scheduled Scale Out 38 Friday, November 15, 13
  93. 93. Other Crucial Design Patterns •Scheduled Scale Out •Clone Server 38 Friday, November 15, 13
  94. 94. Traffic Pattern Normal traffic 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 39 Friday, November 15, 13
  95. 95. Traffic Pattern Normal traffic 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 39 Friday, November 15, 13
  96. 96. Bursty traffic for which Auto-Scale fails TV commercial causes a spike traffic. 40 Friday, November 15, 13
  97. 97. Bursty traffic for which Auto-Scale fails TV commercial causes a spike traffic. 30x traffic # of servers 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 40 Friday, November 15, 13
  98. 98. Bursty traffic for which Auto-Scale fails TV commercial causes a spike traffic. 10min 30x traffic # of servers 20:38 0 20:40 1 2 20:43 3 4 5 20:45 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 40 Friday, November 15, 13
  99. 99. Unexpected Traffic Scheduled –TV Commercial –Introduction by TV program fortunately pre-scheduled! Suddenly happened –Earthquake –New Feature Release burst burst, but gradually tapering 41 Friday, November 15, 13
  100. 100. Prepare for Launching Servers Refresh AMI every day –VR model has been changed everyday Launch AMI for bursty traffic –10 min - 30 min to launch •Loading VR model is taking time.... 42 Friday, November 15, 13
  101. 101. More intelligence 24hours 43 Friday, November 15, 13
  102. 102. More intelligence 24hours Optimized server deployment using Elastic Map Reduce (EMR) for pattern analysis 43 Friday, November 15, 13
  103. 103. Moved twice • Apr 2012 –Start to plan for moving-out to AWS • Jun 2012 –Moved to AWS Northern California • Tokyo did not have enough instances at that time • Deploy mixture instance types (c1.xlarge and m2.4xlarge) • Sep 2012 –Moved to AWS Tokyo • Unified to m2.4xlarge –Improved latency Friday, November 15, 13 44
  104. 104. October, 2012 “Status Quo” DOCOMO “99.9999%” Shrine Skunkworks Temple Office of Inspirations (not ‘Wannabe’ anymore) 45 Friday, November 15, 13
  105. 105. October, 2012 Kid, your game is over. “Status Quo” DOCOMO “99.9999%” Shrine Skunkworks Temple Office of Inspirations (not ‘Wannabe’ anymore) 45 Friday, November 15, 13
  106. 106. October, 2012 Use our stable system. Kid, your game is over. “Status Quo” DOCOMO “99.9999%” Shrine Skunkworks Temple Office of Inspirations (not ‘Wannabe’ anymore) 45 Friday, November 15, 13
  107. 107. October, 2012 Use our stable system. Kid, your game is over. Move on quickly! “Status Quo” DOCOMO “99.9999%” Shrine Skunkworks Temple Office of Inspirations (not ‘Wannabe’ anymore) 45 Friday, November 15, 13
  108. 108. October, 2012 Catch me if you can SCALE! Use our stable system. Kid, your game is over. Move on quickly! “Status Quo” DOCOMO “99.9999%” Shrine Skunkworks Temple Office of Inspirations (not ‘Wannabe’ anymore) 45 Friday, November 15, 13
  109. 109. In DOCOMO Skunkworks (2013) 46 Friday, November 15, 13
  110. 110. In DOCOMO Skunkworks (2013) They are SLOW and pricy. Keep Restless improvement and our leanstartup culture. 46 Friday, November 15, 13
  111. 111. In DOCOMO Skunkworks (2013) They are SLOW and pricy. Keep Restless improvement and our leanstartup culture. Yep, let’s continue our system improvement on the fly. 46 Friday, November 15, 13
  112. 112. In DOCOMO Skunkworks (2013) Hey, leader, how about multi-lingual translation services? They are SLOW and pricy. Keep Restless improvement and our leanstartup culture. Yep, let’s continue our system improvement on the fly. 46 Friday, November 15, 13
  113. 113. In DOCOMO Skunkworks (2013) Hey, leader, how about multi-lingual translation services? They are SLOW and pricy. Keep Restless improvement and our leanstartup culture. Yep, let’s continue our system improvement on the fly. 46 Friday, November 15, 13
  114. 114. Now, we have … 17-Nov-16 27-Nov-16 7-Dec-16 17-Dec-16 27-Dec-16 6-Jan-17 16-Jan-17 26-Jan-17 5-Feb-17 15-Feb-17 25-Feb-17 7-Mar-17 17-Mar-17 27-Mar-17 6-Apr-17 16-Apr-17 26-Apr-17 6-May-17 16-May-17 26-May-17 5-Jun-17 15-Jun-17 25-Jun-17 17-Nov-12 5-Jul-17 15-Jul-17 25-Jul-17 4-Aug-17 14-Aug-17 24-Aug-17 Number of Speech • 10 million terminals pre-installed • More than 4M distinct user accesses • 1M accesses / day 47 Friday, November 15, 13
  115. 115. Other Tips • Instance Management ✓Simple DB as a key-value store • Log management ✓Enormous logs from servers ✓Direct write to S3 • Multi-AZ ✓HA and act-act configuration 48 Friday, November 15, 13
  116. 116. Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’ 49 Friday, November 15, 13
  117. 117. Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’ in changing our development style to 49 Friday, November 15, 13
  118. 118. Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’ in changing our development style to “Deploy first, think later,” 49 Friday, November 15, 13
  119. 119. Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’ in changing our development style to “Deploy first, think later,” 49 Friday, November 15, 13
  120. 120. Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’ in changing our development style to “Deploy first, think later,” in bringing us a high performance culture, and 49 Friday, November 15, 13
  121. 121. Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’ in changing our development style to “Deploy first, think later,” in bringing us a high performance culture, and 49 Friday, November 15, 13
  122. 122. Thanks to the people who gave great lessons to ‘DOCOMO Skunkworks.’ in changing our development style to “Deploy first, think later,” in bringing us a high performance culture, and in becoming “Cloud Natives.” 49 Friday, November 15, 13
  123. 123. Please give us your feedback on this presentation MBL202 As a thank you, we will select prize winners daily for completed surveys! Thank You 50 Friday, November 15, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×