From Google Cloud services to big data system

1. Big Data In Action
Google IO Extended 2016

2. Tu Pham
Google Cloud Expert
• Co-Founder of DYNO (Online email marketing SaaS)
• Data engineer at MOG Corp

3. For the past 17 years, Google
has been building out the
world’s fastest, most powerful,
highest quality cloud
infrastructure on the planet.
Images by ConnieZhou

4. Google Cloud Platform is built on
the s a m e infrastructure that
powers Google.
Images by Connie Zhou

5. Google’s Platform
“[Google's] ability to build, organize, and
operate a huge network of servers and
fiber- optic cables with an efficiency and
speed that rocks physics on its heels.
This is whatmakes Google Google: its physical
network, its thousands of fiber miles, and
those many thousands of servers that, in
aggregate, add up to the mother of all
clouds.”
-
Wired

6. 77
Peering locations

7. Yes, We Can Power that
Web Mobile Storage & Database
Big Data Highly Scalable System Data Mining
Cloud Platform

8. Google CloudPlatform
Organize the world’s
information and make it
universally accessible and
useful.
Google’s Mission
2
“

9. Google CloudPlatform 5
Source: Boston Consulting Group:
The Mobile Revolution: How Mobile Technologies Drive a Trillion-DollarImpact
IDC,2015
By 2020, there will be 8 Billion connected smart phones
— 2X more than today.
And 32 Billion connected “IOT”devices
—6X more thantoday.

10. Exploring the Cloud
IaaS
Infrastructure-as-a-
Service
PaaS
Platform-as-a-
Service
SaaS
Software-as-a-
Service
Google Cloud
Platform
Cloud Platform

11. Google Compute Engine
Cloud Platform
• Flexible Infrastructure
• Customer VM Size
• Online Disk Resizing
• Network
• Internal Network
• Firewall
• Load Balancing
• External Ip Address
• Billing
• Sustained Usage Discounts
• Preemptible VM

12. App Engine
• Fully Managed Platform
• Popular Programming Language Support
• Flexible and Scalable Application Storage
• Auto-scaling
• Versioning and Traffic Splitting
• Local Developer Tools
• Third-party Frameworks and Extensions
Cloud Platform

13. • Global Presence
• Flexible Delivery Options
• Pull
• Push
• Data Reliability
• Flow Control
• Data Security And Protection
Cloud Platform
Pub Sub

14. • Reliable & Consistency
Processing
• Unified Programing Model
• Intelligence Work Scheduling
• Auto Scaling
• Monitoring
• Open Source
Cloud Platform
Cloud Data Flow

15. • Versioning
• Static Sites
• Resumable Transfers
• Object Change Notifications
• TB scale
Cloud Platform
Cloud Storage

16. Cloud SQL
• Fully managed
• Ease of Use
• Highly Reliable
• Flexible Charging
• Security, Availability, Durability
• Easy Migration & Data
Portability
• Optimized Mysql versions
Cloud Platform

17. BigQuery
• Fully Managed Big Data Analytics Service
• Support SQL
• Fast
• Scalable
• Flexible and Familiar
• Security and Reliability
Cloud Platform

18. Data Proc
• Includes
• Apache Hadoop
• Apache Pig
• Apache Hive
• Apache Spark
• Fast And Scalable Data Processing
• Flexible Virtual Machines
• Resizable Cluster
Cloud Platform

19. Datalab
• Powerful Data Exploration
• Scalable
• Data Management
• Visualization
• Open Source (Jupyter)
Cloud Platform

20. 10+ Years of Tackling Big Data Problems
Google CloudPlatform 13
Google
Papers
20082002 2004 2006 2010 2012 2014 2015
GFS
Map
Reduce
Flume
Java
Millwheel
Open
Source
2005
Google
Cloud
Products BigQuery Pub/Sub Dataflow Bigtable
BigTable Dremel PubSub
Apache
Beam
Tensorflow

21. Google’s Data Services for everyone

22. Confidential +
Proprietary
Storage andDatabases
CloudStorage
The Google Cloud data toolbox
CloudSQL
CloudBigtable
CloudDatastore
Big Data andAnalytics
BigQuery
CloudPub
Sub
CloudDataflow
CloudDataproc
CloudDatalab
Machine Learning
Cloud Machine
Learning
Cloud TranslateAPI
Cloud VisionAPI
Cloud SpeechAPI

23. A common configuration: draw conclusions
CloudDatalab
Events, metrics,
etc.
Stream
Visualization and BI
Raw logs, files,
assets, Google
Analytics data etc. Co-workers
Batch
Batch
B C Applications and
A Reports
Confidential +Proprietary
A serverless big data stack that
scales automatically

24. Confidential &ProprietaryGoogle CloudPlatform 18
Complexities of Big Data Processing
Programming
Resource
provisioning
Performance
tuning
Monitoring
Reliability
Deployment&
configuration
Handling
growingscale
Utilization
improvement
s
Time to
Understanding
Typical Big Data
Processing

25. Confidential &ProprietaryGoogle CloudPlatform 19
Spend Time on ‘What’ not ‘How’
Time toUnderstanding
Big Data
Processing with
Google Cloud
Platform
Programming
More time to dig
into your data

26. Big Data Lifecycle
Cloud Logs
Google App
Engine
Google Analytics
Premium
Cloud Pub/Sub
BigQuery Storage
(Tables)
Cloud Bigtable
(NoSQL)
Cloud Storage
(Files)
Cloud Dataflow
BigQuery Analytics
(SQL)
Analyze
Batch
Capture Process Store
Stream
Cloud
Monitoring
Real-time analytics
Cloud Dataflow
CloudML
Real-time
dashboard
Real-time
alerts
Use
Data
Scientists
Analysts
Smart
apps
Catalog & Data Lifecycle Automation
Cloud
Datalab
Cloud Dataproc
Data Studio

27. Confidential &ProprietaryGoogle CloudPlatform 21
Emerging Big Data Challenges
Real-time
data ingestionBatch or streaming?
Analytics at the
speed of thought

28. Batch orStreaming?
Why do you have to choose?
Breakthrough #1

29. Google CloudPlatform Confidential &Proprietary
We don’t really use MapReduceanymore
UrsHölzle
SVP Technical
Infrastructure Google
“ ”23

30. Confidential +
Proprietary
A common configuration: capturing input
Cloud Pub/Sub
Reliable, many-to-many, asynchronous messaging
CloudStorage
Powerful, simple and cost-effective objectstorage
Raw logs, files,
assets, Google
Analytics data etc.
Events, metrics,
etc.

31. Confidential +
Proprietary
A common configuration: process and transform
Events, metrics,
etc.
Cloud Dataflow
Data processing engine for
batch and streamprocessing
Stream
Batch
Raw logs, files,
assets, Google
Analytics data etc.

32. Confidential +
Proprietary
A common configuration: process and transform
Events, metrics,
etc.
Cloud Dataflow
Data processing engine for
batch and streamprocessing
Stream
Batch
Batch
CloudDataproc
Managed Spark andHadoop
Raw logs, files,
assets, Google
Analytics data etc.

33. Confidential +
Proprietary
A common configuration: analyze and store
Events, metrics,
etc.
Stream
Batch
Batch
Raw logs, files,
assets, Google
Analytics data etc.

34. Confidential +
Proprietary
A common configuration: draw conclusions
Events, metrics,
etc.
Stream
Batch
Applications and
Reports
CloudDatalab
Visualization and BI
Co-workers
Batch
B C
A
Raw logs, files,
assets, Google
Analytics data etc.

35. Real-time data ingestion
(and at scale)
Breakthrough #2

36. Google confidential │ Do not distribute
Overview:
Data to process: Data in the Consolidated Audit Trail
(CAT). A data repository of all equities and options orders,
quotes, and events
Challenges:
How to process the CAT and organize 100 billion
market events into an “order lifecycle” in a 4 hour
window
Store 6 years (~30PB) of data
Cloud Bigtable to process and run queries
and tolerate volume increases
6BILLION
MARKET EVENTS
WRITTEN PER HOUR
1.7 GIGs
6TBs
10 BN
WRITTEN
PER HOURBURSTS
1.7 GIGABYTES
PER SECOND
10TERABYTES
PER HOUR

37. Analytics at the speedof
thought
(and at scale)
Breakthrough #3

38. Building what’s next 33
Scales automatically
No setup or administration
Stream up to 100,000 rowsp/sec
Easily integrates with third-partysoftware
Google BigQuery
makes complex data analysis simple

39. Confidential +
Proprietary
Google BigQuery Performance Example ?
Running an inefficient regular expression over 100 billion rowsin
less than 60 seconds
Source: https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-bigquery-
query

40. Google BigQuery
The Power of Google Dremel for everyone
Storage Compute
Fast Ingest
Query
Terabit Network

41. 1000-core Hadoop Cluster
= 2.5 hours
Before
Making ad hocQueries
with BigQuery <5min
After
● 500+ Games
● Hundreds of Analysts
● Terabytes of Data Daily

43. “Right at the start of the partnership we were
able to reduce time to insight from 96 hours to
30 minutes by using BigQuery, allowing us to
react in real time to customer needs and provide
better service..”
GarySanders
Head of the bank's digital analyticsfunction
https://www.finextra.com/newsarticle/28566/lloyds-partners-google-on-data-analytics

44. Connect and Stay Informed
Google Cloud Platform WebSite
https://cloud.google.com
Google Cloud Platform Blog
https://googlecloudplatform.blogspot.com
Google+
https://plus.google.com/+GoogleCloudPlatform
Twitter
https://twitter.com/GoogleCloud

45. Demo: Tracking Pixel
• Compute Engine
• Instance Template
• Access Scope
• Instance Group
• Network
• Firewall
• Load Balancing
• Stackdriver
• BigQuery
• Data Lab
• App Engine
• Source Repositories

46. Bring it all together

47. cloud.google.com
Images byConnie
Zhou

48. Thank you!
Questions?
● tu@dyno.me
● @phamptu