Video from presentation: https://youtu.be/SoSZdI2lMVQ Processing vast amounts of data in the cloud has long been a nightmare not just for data analysts but also budget owners. We believe that migrating your data engineering workloads to can be beneficial, if you keep in mind some basic architectural principles. Teams processing big data in the cloud should understand and leverage its key attribute. Flexibility.The goal of our keynote is to share our experience and key learnings on how to fully utilize the power that the cloud offers and not go broke. This could be useful for both startups, but also large corporation as we will show examples of how to dramatically lower the cost of infrastructure. Speaker: Johnson Darkwah, Big Data Solution Architect at Gauss Algorithmic, https://www.linkedin.com/in/johnson-darkwah-7ba76511/

1. Effective use of cloud resources
for Data Engineering
A guide to Cloud Computing & Big Data
12/2017

2. Obsessed With Creating Value From Data

3. Expert in Big Data Architecture
Certification and Recognition
from Cloudera
Specialization in Telco Data
Monetization
Johnson Darkwah

4. Where are you running your Big Data ?
The vast majority has been
running on premise, on
private infrastructure

5. 33% of enterprises will take their data
lakes off life support.
Forrester Predictions 2018: The Honeymoon
For AI Is Over

6. Initial Cost of a Production
Big Data Management
Platform running On
Premise
200,000 EUR
5,092,000 CZK

7. Challenges with the Cloud
• Security remains a concern
for early adopters
• Experienced companies
focus on optimizing cost
• Data integration poses a
challenge
• Lack of resources

8. Why use the cloud today for analytics
It’s never been
cheaper to process
and store data in the
cloud
It’s never been
easier to manage
cloud services
Cost Flexibility

9. Master
m4.xlarge
1x 1TB EBS
Worker
m4.2xlarge
4x 1TB EBS
Worker
m4.2xlarge
4x 1TB EBS
Worker
m4.2xlarge
4x 1TB EBS
Worker
m4.2xlarge
4x 1TB EBS
Worker
m4.2xlarge
4x 1TB EBS
Data processing power of 32 CPU cores, 120GB
RAM and the ability to store up to 6,5 TB of data.
When you treat cloud the same way
as on-premise

10. Monthly Costs on AWS
AWS EC2 (Europe Central)
- Compute: 1932.48 USD
- EBS Volumes: 1134.00 USD
$ 3,066.49 USD
65,371 CZK
AWS EC2 (Europe Central)
- Compute: 8725.44 USD
- EBS Volumes: 4860.00 USD
$ 13,585.44 USD
289,605 CZK
Reasonable minimum
spec
Comparable to
production on-prem.
https://calculator.s3.amazonaws.com/index.html

11. ü Utilize Object Stores
ü Pay only for what we really use
ü Get discounts
ü Optimize the Data Architecture
ü Automate Everything
Cloud Data Engineering Checklist

12. The cost of storing a GB in a Object Store (S3 or ADLS) is 2/3 of the coston HDFS

13. ü Utilize Object Stores
ü Pay only for what we really use
ü Get discounts
ü Optimize the Data Architecture
ü Automate Everything
Cloud Data Engineering Checklist

14. Most companies do not need a full scale cluster 24/7.
CPU Usage %

15. ü Utilize Object Stores
ü Pay only for what we really use
ü Get discounts
ü Optimize the Data Architecture
ü Automate Everything
Cloud Data Engineering Checklist

16. On-Demand Reserved Spot
Similar models are now also available on Azure and GCP.
Pay for compute capacity
by the hour with no long-
term commitments
Make 1-3 years
commitment and receive a
significant discount on
instance and storage
pricing.
Bid for unused capacity,
charged at a Spot price
that fluctuates based on
supply and demand.
Full Price Over 50% off Over 80% off

17. Spot Average Prices

18. ü Utilize Object Stores
ü Pay only for what we really use
ü Get discounts
ü Optimize the Data Architecture
ü Automate Everything
Cloud Data Engineering Checklist

19. Realtime System 24x7 S3/ADLS M W W W W W W
SQL Engine
Terraform / Cloudera Altus
Existing Enterprise
Logs, CRM, RDBMS
Gauss
Data Tool
Clicks IoT Trans.
Temp cluster
Client Job
ML, SQL, Enrichment, …
Spot
On-premise
Cloud
RIOSOD

20. Show me the money …
AWS EC2 (Europe Central)
Realtime System (RI – 24x7)
- Compute (3x): 352.52 USD
- EBS Vol. (3x1TB): 54.00 USD
Temp
- Compute (OD 8x7):
- 1x m4.xlarge: 58.56 USD
- 3x m4.2xlarge: 351.36 USD
- EBS Vol. (9x1TB): 162.00 USD
$ 1,165.78 USD
24,852.10 CZK
Spot support (3x – m4.2xlarge - $0.2/h)
- Compute(): 34.34 USD
- EBS Vol. (9x1TB): 162.00 USD
https://calculator.s3.amazonaws.com/index.html
8 Hours Compute Daily

21. ü Utilize Object Stores
ü Pay only for what we really use
ü Get discounts
ü Optimize the Data Architecture
ü Automate Everything
Cloud Data Engineering Checklist

22. Terraform
• Enables simple management and automation
of cloud environments
• AWS, Azure, GCP, OpenStack, …
• Many provisioning options
# Configure the AWS Provider
provider "aws" {
access_key = "${var.aws_access_key}"
secret_key = "${var.aws_secret_key}"
region = "us-east-1"
}
# Create a datanode
resource "aws_instance" ”datanode" {
# ...
}

23. Cloudera Altus
• Enterprise ready & fully managed
cluster
• Spark, MRv2 and Hive supported
• Support for spot instances
• Uses S3 or ADLS
• Clusters can run only when
executing jobs
• Custom pre-built image (AMI)
support
• Per hour pricing with support
included

24. ü Utilize Object Stores
ü Pay only for what we really use
ü Get discounts
ü Optimize the Data Architecture
ü Automate Everything
Cloud Data Engineering Checklist

25. Twitter: @darkwahj
LinkedIn: Johnson Darkwah
Do more with data
& save money.
Questions ?
www.gaussalgo.com