Big Data & Data Engineering

1. © 2017 Google Inc. All rights reserved.
Using Google Cloud Platform
for creating Big Data Analysis
and IoT solutions
Mykola Murha,
PhD, Google Cloud Certified
Professional Data Engineer,
Data Scientist at Techsvit

2. © 2018 Google LLC. All rights reserved.
Вanking transactions
IoT telemetry
Data about interactions with clients
Google Analytics
Sources of BIG Data:

3. © 2017 Google Inc. All rights reserved.© 2017 Google Inc. All rights reserved.
Main problems for Big Data
Science:
… How to collect and consolidate data?
… How and where store data?
… How to select, analyze and model data?

4. © 2017 Google Inc. All rights reserved.
Cloud iOT Core Cloud PubSub Cloud Function
Different connectors
Different
Data
Sources
Data collection and
Google Cloud IoT Core:
Google Cloud IoT Core in conjunction with Cloud PubSub and Cloud Function allow dynamically
and safety collect data from different IoT devices. It allows to use MQTT and HTTP protocols to
communicate with ones

5. © 2017 Google Inc. All rights reserved.
Persistent Disk
Cloud analogous for
hard drive allows to
simply configure
volume and type
(traditional HDD,
SDD)
Cloud Storage
Storage for files and
static content in
Google Cloud could
be published. Simple
migration interface
with AWS S3
Data Storage:
Cloud Filestore
Cloud NFS-server
which can be simply
connected to Cloud
Virtual Machines

6. © 2017 Google Inc. All rights reserved.
Data Storage:
Cloud Bigtable
Columnar DBMS
fits very well for
time series
storage and the
same processes
Cloud Datastore
NoSQL DBMS allows to
add dynamically
columns to the entities,
using standard SQL at
the same time.
Cloud Firestore
Scalable DBMS fits
well for mobile and
web-applications

7. © 2017 Google Inc. All rights reserved.
Cloud Memorystore
GCP Redis
Cloud Spanner
Dynamically scalable
relational DBMS
Cloud SQL
Cloud MySQL or
PostgreSQL
Cloud BigQuery
Columnar DBMS allows
to organize DWH
from different
heterogeneous data
sources.
Data Storage:

8. Confidential & ProprietaryGoogle Cloud Platform 8
Cloud Datalab
Cloud Jupiter
notebook
Cloud Dataprep
Google Cloud tool
allows to automate
data pre-processing
and connect users to
the process
Cloud Dataproc
GCP solution for
task to be solved
with Hadoop and
Spark, saving
syntax at the same
time
Google Data Studio
Tool got on-line report
automation.
Cloud Dataflow
Streaming and batch
Big Data processing
based on Google
Cloud. Based on
Apache Beam
Data Analysis:

9. © 2017 Google Inc. All rights reserved.
APIs and tools for text,
video, images, speech
processing, translation
Cloud AutoML Natural Language
Cloud AutoML Translate
Cloud AutoML Vision
Cloud Natural Language
Cloud Speech-To-Text
Cloud Text-To-Speech
Cloud Translation API
Cloud Video Intelligence
Cloud Vision API
Dialogflow Enterprise Edition
Cloud Machine
Learning Engine
Google Cloud tools allows to
concentrate only on modelling
process but it is possible to adjust
some technical elements.
Data Modelling:

10. How to simply deploy
Hadoop and Spark:
Cloud Dataproc
Cloud Storage
Cloud Compute Engine

11. © 2017 Google Inc. All rights reserved.
Heterogeneous DWH
using BigQuery:
Different
Data Sources
Cloud Storage Cloud BigQuery
Google Sheets
(G Suite in general)

12. © 2017 Google Inc. All rights reserved.
Different
Data Sources
On-line reporting using
Google Data Studio:
Huge amount of
standard connectors
Google
Data Studio

13. © 2017 Google Inc. All rights reserved.
Different
Data Sources
Google Cloud Dataflow:
Data processing
pipeline
Different
Data Sources
Cloud Dataflow

14. © 2017 Google Inc. All rights reserved.
Machine Learning Engine:
Cloud Storage
Machine Learning
Engine
Cloud Datalab

15. IoT Solution based on
Google Cloud Platform:
{"payloadMetaData":{"applicationMetaData":{"id":{"entityType":"APPLICATION","id":"b47ad870-
797e-11e8-9597-99f29e0f7a38"},"customerId":{"entityType":"CUSTOMER","id":"624e6d60-797d-
11e8-9597-99f29e0f7a38"},"subCustomerId":null,"name":"Home
Sensors"},"gatewayMetaDataList":[{"id":{"entityType":"GATEWAY","id":"f2243340-956e-11e8-822e-
99f29e0f7a38"},"name":"Pico
1746D0011","mac":"647FDAFFFE005005","latitude":null,"longitude":null,"altitude":null,"rxInfo":{"cha
nnel":2,"codeRate":"4/5","crcStatus":1,"dataRate":{"modulation":"LORA","spreadFactor":12,"bandwid
th":125},"frequency":868500000,"loRaSNR":9.0,"mac":"647fdafffe005005","rfChain":1,"rssi":-
51,"size":16,"time":"2019-01-
16T16:01:48Z","timestamp":2787174116,"rsig":null,"antenna":0}}],"deviceMetaData":{"id":{"entityTy
pe":"DEVICE","id":"15893ed0-ef23-11e8-8fef-
3d9ffe0fa0e3"},"name":"1839N0036","deviceClass":"CLASS_A","deviceEUI":"647FDA000000107D","
appEUI":"647FDA8010000100"},"adr":true,"fcount":3043,"fport":10},"payload":{"motionDetected":tru
e}}
HTTP
Google Data Studio Cloud Storage Cloud Functions Cloud PubSub
Cloud IoT Core

16. © 2017 Google Inc. All rights reserved.
Online analytics on
Google Cloud Platform:
Cloud Storage Google Data Studio
Google BigQuery

17. Using different types of Google Cloud Storages:

18. © 2017 Google Inc. All rights reserved.© 2017 Google Inc. All rights reserved.
What Techsvit
can do for you:
… Organizing Google Cloud Platform account with
discount.
… Consulting concerning Big Data storage and
processing basing on Google Cloud Platform.
… Organize Cloud work environment for Data Scientist
where he or she thing about goals and features but not
about technical nuances.
… Build and implement Big Data
(machine learning) models into you processes.

19. © 2017 Google Inc. All rights reserved.
Thank you!
techsvit.eu
+380 (44) 33 444 33
mm@techsvit.eu