Want to take advantage of machine learning without building and training your own models? The Google Cloud Vision and Speech APIs expose the machine learning functionality behind Google Photos, Google Images, and the speech recognition in “Ok, Google.” Developers can now build these features into their apps with a simple REST API call. This deck gives an overview of Google Cloud Vision API, Cloud Speech API, and the Cloud Natural Language API.

1. Puppies or muffins?
Easily leverage machine learning in your apps
Sara Robinson
@SRobTweets
Bret McGowen
@bretmcg

2. 2@SRobTweets @bretmcg
Who are we?
Developer Advocate, Google Cloud Platform
Sara Robinson / @SRobTweets
● New York, NY
● Swift fan (Taylor and language)
● Harry Potter aficionado
Developer Advocate, Google Cloud Platform
Bret McGowen / @bretmcg
● New York, NY
● U2 fan (band and plane)
● Lord of the Rings aficionado

3. What we’ll cover
01
02
03
04
05
A (very) brief overview of machine learning
Machine learning at Google
Vision API
Speech API
Natural Language API

4. 01 A (very) brief overview of machine learning

5. 5@SRobTweets @bretmcg
Machine Learning, Then

6. 6@SRobTweets @bretmcg
Machine Learning, Now

7. 7@SRobTweets @bretmcg

8. 02 Machine Learning at Google

13. Combined vision and translation

15. 15@SRobTweets @bretmcg
Image Search

16. 16@SRobTweets @bretmcg
Google Photos

17. Google Cloud Platform 17
Doesn't seem THAT hard...

18. Images: Wikimedia
Source: https://commons.wikimedia.org/wiki/File:Red_Apple.jpg
https://en.wikipedia.org/wiki/Orange_(fruit)#/media/File:Orange-Whole-%26-Split.jpg

19. Google Cloud Platform 19
What if I'm not a machine learning expert?

20. 20@SRobTweets @bretmcg
The Machine Learning Spectrum
TensorFlow Cloud Machine Learning Machine Learning APIs
BYOML skills
(Friendly Machine Learning)
Pre-packaged ML

21. 02 The Cloud Vision API
Complex image detection with a simple REST request

23. 03 Making an API request

24. Making a request
{
"requests":[
{
"image": {
"content": "base64ImageString"
// Alternatively, you can pass a Google Cloud Storage url here
},
"features": [
{
"type": "LABEL_DETECTION",
"maxResults": 10
},
{
"type": "FACE_DETECTION",
"maxResults": 10
},
// More feature detection types...
]
}
]

25. Google Cloud Platform 25
Let’s see some JSON responses

26. 26
{
"labelAnnotations" : [
{
"mid" : "/m/01wydv",
"score" : 0.92442685,
"description" : "beignet"
},
{
"mid" : "/m/0270h",
"score" : 0.90845567,
"description" : "dessert"
},
{
"mid" : "/m/033nb2",
"score" : 0.74553984,
"description" : "profiterole"
},
{
"mid" : "/m/01dk8s",
"score" : 0.71415579,
"description" : "powdered sugar"
}
]
}
Label Detection
26

27. "landmarkAnnotations": [
{
"mid": "/m/0c7ln",
"description": "Navy Pier",
"score": 36,
"boundingPoly": {
"vertices": [
{
"x": 275,
"y": 102
}, //...
]
},
"locations": [
{
"latLng": {
"latitude": 41.888685,
"longitude": -87.601311
}
}
], //...
},
{
"mid": "/m/01_d4",
"description": "Chicago",
"score": 31,
Landmark Detection
27

28. 28
...
"itemListElement": [
{
"@type": "EntitySearchResult",
"result": {
"@id": "kg:/m/0c7ln",
"name": "Navy Pier",
"@type": [
"Thing", "Place", "LandmarksOrHistoricalBuildings",
"TouristAttraction"
],
...
"detailedDescription": {
"articleBody": "Navy Pier is a 3,300-foot-long
pier on the Chicago shoreline of Lake Michigan. It
is located in the Streeterville neighborhood of
the Near North Side community area.",
"url": "http://en.wikipedia.org/wiki/Navy_Pier"
...
Knowledge Graph sidebar
GET https://kgsearch.googleapis.com/v1/entities:search?ids=%2Fm%2F0b__kbm&key={API_KEY}

29. 29
"textAnnotations": [
{
"locale": "en",
"description": "U.S. COAST GUARD AUXILIARYn242039n",
"boundingPoly": {
"vertices": [
{
"x": 429,
"y": 307
},
{
"x": 1178,
"y": 307
},
{
"x": 1178,
"y": 770
},
{
"x": 429,
"y": 770
}
]
}
},
// ...
]
Text Detection

30. 30
"faceAnnotations" : [
{
"headwearLikelihood" : "VERY_LIKELY",
"surpriseLikelihood" : "VERY_UNLIKELY",
"rollAngle" : 2.8030474,
"angerLikelihood" : "VERY_UNLIKELY",
"landmarks" : [
{
"type" : "LEFT_EYE",
"position" : {
"x" : 221.60617,
"y" : 638.263,
"z" : 0.0017568493
}
},
...
],
"boundingPoly" : {
"vertices" : [
{
"x" : 89,
"y" : 436
},
...
Face Detection
"detectionConfidence" : 0.98838496,
"joyLikelihood" : "VERY_LIKELY",
"panAngle" : -1.0822374,
"sorrowLikelihood" : "VERY_UNLIKELY",
"tiltAngle" : -2.5003448,
"underExposedLikelihood" : "VERY_UNLIKELY",
"blurredLikelihood" : "VERY_UNLIKELY"

31. 31
"faceAnnotations" : [
{
"headwearLikelihood" : "VERY_UNLIKELY",
"surpriseLikelihood" : "VERY_UNLIKELY",
rollAngle" : -4.6490049,
"angerLikelihood" : "VERY_UNLIKELY",
"landmarks" : [
{
"type" : "LEFT_EYE",
"position" : {
"x" : 691.97974,
"y" : 373.11096,
"z" : 0.000037421443
}
},
...
],
"boundingPoly" : {
"vertices" : [
{
"x" : 743,
"y" : 449
},
...
Face Detection
"detectionConfidence" : 0.93568963,
"joyLikelihood" : "VERY_LIKELY",
"panAngle" : 4.150538,
"sorrowLikelihood" : "VERY_UNLIKELY",
"tiltAngle" : -19.377356,
"underExposedLikelihood" : "VERY_UNLIKELY",
"blurredLikelihood" : "VERY_UNLIKELY"

32. 32
"labelAnnotations" : [
{
"mid" : "/m/01fklc",
"score" : 0.9337945,
"description" : "pink"
},
{
"mid" : "/m/09g5pq",
"score" : 0.83878618,
"description" : "people"
},
{
"mid" : "/m/017ftj",
"score" : 0.71847415,
"description" : "sunglasses"
},
{
"mid" : "/m/019nj4",
"score" : 0.69381392,
"description" : "smile"
}
]
Putting it all together: face + label + landmark
"landmarkAnnotations" : [
{
"boundingPoly" : {
"vertices" : [
{
"x" : 153,
"y" : 64
},
...
]
},
"mid" : "/m/0c7zy",
"score" : 0.56636304,
"description" : "Petra",
"locations" : [
{
"latLng" : {
"longitude" : 35.449361,
"latitude" : 30.323975
}
}
]
}
]

33. 33
"safeSearchAnnotation" : {
"spoof" : "VERY_UNLIKELY",
"medical" : "VERY_UNLIKELY",
"adult" : "VERY_UNLIKELY",
"violence" : "VERY_UNLIKELY"
}
But wait...is it appropriate?

34. 04 Live Demo

35. 03 The Speech API
Speech to text transcription in over 80 languages

36. 36@SRobTweets @bretmcg
What can I do with the Speech API?
● Speech to text transcription in over 80 languages
● Supports streaming and non-streaming recognition
● Filters inappropriate content

37. 37@SRobTweets @bretmcg
Translation Response
"responses": [{
"results": [{
"alternatives": [
{
"transcript": "how old is the Brooklyn Bridge",
"confidence": 0.987629
}],
"isFinal": true
}
]
}]

38. 38@SRobTweets @bretmcg
Let’s make a recording!
1. Make a recording using SoX, a command line utility for audio files
2. Base64 encode the recording
3. Build our API request in a JSON file
4. Send the JSON request to the Speech API
Bash script for this: bit.ly/speech-request-script

39. 04 Cloud Natural Language API
Perform sentiment analysis and entity recognition on text

40. 40@SRobTweets @bretmcg
What can I do with the Natural Language API?
Three methods:
1. Analyze entities - The Cubs are an MLB team from Chicago
2. Analyze sentiment - I love Chicago
3. Analyze syntax - Michelle Obama is married to Barack Obama

41. 41@SRobTweets @bretmcg
Analyze Entities
Chicago is
The Wrigley Building, Chicago is
The Union Stockyard, Chicago is
One town that won't let you down
It's my kind of town
-- Frank Sinatra

42. 42@SRobTweets @bretmcg
Analyze Entities
Chicago is
The Wrigley Building, Chicago is
The Union Stockyard, Chicago is
One town that won't let you down
It's my kind of town
-- Frank Sinatra

43. 43@SRobTweets @bretmcg
Analyze Entities
Chicago is
The Wrigley Building, Chicago is
The Union Stockyard, Chicago is
One town that won't let you down
It's my kind of town
-- Frank Sinatra

44. 44@SRobTweets @bretmcg
Analyze Sentiment
Chicago is the best city in the world.
{
“documentSentiment”: {
“polarity”: 1,
“magnitude”: 0.8
}
}

45. 45@SRobTweets @bretmcg
Analyze Syntax
“The Chicago Cubs are an American professional baseball team based in
Chicago, Illinois.”

46. 46@SRobTweets @bretmcg
Analyze Syntax

47. 47@SRobTweets @bretmcg
Let’s see a demo!
Twitter Streaming NL API BigQuery
bit.ly/nl-olympics

48. 48@SRobTweets @bretmcg
APIs we covered
● Vision: cloud.google.com/vision
● Speech: cloud.google.com/speech
● Natural Language: cloud.google.com/natural-language
Related APIs:
● Translate: cloud.google.com/translate
● Prediction: cloud.google.com/prediction
● Knowledge Graph API: developers.google.com/knowledge-graph

49. Thank You
Sara Robinson
@SRobTweets
Bret McGowen
@bretmcg