BI Meets Serverless on Cloud - How to build your BI pipeline within minutes

1. BI Meets
Serverless on
Cloud
Build your BI Stack on AWS
within minutes
07.12.2018
Gökberk Keskin

2. Outline
Introduction
Solution Architecture
- Storage
- Serverless Database
- Application Server
- BI/Reporting Service
Authentication & Authorization
Notes and Tips about Billing
Build and Demo
Building Your Instance Tips
Compliance - Data Privacy
Questions
References

3. Introduction
Cloud Storage Serverless
Database
Cloud
Computing
Platform
BI / Reporting
Service

4. Solution Architecture

5. Solution Architecture
Upload
Input
Data
Use
Data
Use as
Reporting Tool
End User
Use as Data
Source
Storage
Serverless
Database
Application
Server

6. Storage
➔ AWS S3 Buckets used as storage service
➔ S3 is an object storage to store/retrieve data
➔ Reduces effort to maintain your storage
➔ Highly scalable and you only pay what you use
➔ Key based object storage. Folder structure is virtual

7. Serverless Database
➔ AWS Athena used as serverless database services for analytics purposes
➔ Athena uses :
◆ Hive to create, drop, and alter tables and partitions
◆ Presto, a distributed SQL engine to run queries
➔ Helps as an interactive query service to analyze data in S3
➔ Able to connect data files including CSV, JSON, Avro or columnar data formats such
as Apache Parquet and ORC
➔ Data is not loaded in Athena. It just provides service to connect data

8. Serverless Database / 2
PrestoDB Overview
CREATE EXTERNAL TABLE `hr_training`(
`year` int,
`organization` string,
`department` string,
`position` string,
`position_count` int,
`planned_position_count` int,
`expense_total` string,
`course_cost` string,
`course_days` int,
`terminations` int,
`internal_hires` int,
`external_hires` int)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://fff-bi-core-data/'
TBLPROPERTIES (
'has_encrypted_data'='false',
'transient_lastDdlTime'='1541250964')
Athena Create Table Syntax

9. Application Server
➔ Linux Ubuntu 18.04 server running on an EC2 Instance
➔ Instance is preconfigured by Redash as Machine Instance
➔ T2.small size instance needed at least to run Redash

10. BI / Reporting Service
➔ Lightweight and generic open source reporting and data visualization tool
➔ Variety of hosting
◆ Open source base self hosted (Linux machine or docker)
◆ Open source base Cloud hosted (Amazon AMI)
◆ Google Cloud hosted
◆ Redash hosted
➔ Variety of supported data integrations

11. Redash Supported Integrations

12. Authentication and Authorization

13. Authentication & Authorization
➔ To authenticate users, generic users are created in redash
LDAP authentication is also possible
➔ AWS Identity Access Management service used for authorization
➔ To access from redash to Athena and S3, user needs to be authorized with
policies
➔ IAM has policies and roles, which can be assigned to users and groups

14. Authentication & Authorization / 2
Redash uses API key generated by IAM.
In IAM, these users/groups have policy to access Athena and S3
Use API
Key
Assign
Policy
Access
IAM

15. Notes and Tips About Billing

16. Notes and Tips About Billing
➔ Elastic nature of EC2 services might cost a bit more than you think
◆ Try to stay in AWS Free Tier as much as possible
Redash requires t2.small instance, which is out of free tier unfortunately
◆ Remember to keep an eye on Billing & Cost Management Dashboard
◆ Create your AWS Budget and create your budget reports to see your expenses
◆ Do not forget to stop instances if you do not use

17. Build and
Demo
AWS S3 + Athena + Redash
To start open source AWS based setup, let’s
follow steps in
https://redash.io/help/open-source/setup#aws

18. Building Instance Tips
➔ In first time to create instance, save your server Amazon private key securely
➔ Convert amazon .pem private key to ppk by PuttyGen to use key in Putty
➔ Design your inbound/outbound security rules carefully, create your own restricted IAM policies
➔ Make sure you select T2.Small to be able to run Redash
➔ In Europe, Athena is only available in eu-west-1 and eu-west-2 regions
➔ If you plan to reuse a stopped EBS backed instance, do not terminate
➔ Make sure you have set your S3 bucket access policy well
e.g no public buckets, no full S3 access
➔ For advanced data security, evaluate using data encryption in your S3 buckets

19. Compliance - Data Privacy

20. Thanks for listening :)
Questions?

21. References
https://aws.amazon.com/s3/faqs/?nc1=h_ls
https://aws.amazon.com/tr/athena/faqs/
https://aws.amazon.com/big-data/what-is-presto/
https://redash.io/product/
https://redash.io/integrations/
https://aws.amazon.com/compliance/data-privacy/
https://aws.amazon.com/blogs/big-data/analyzing-data-in-s3-using-amazon-athena/
https://www.linkedin.com/pulse/why-amazon-athena-great-your-big-data-analytics-needs-ido-amit/