This is the presentation I used for the 2rd Kylin Meetup at Shanghai. I was invited to give a speech about our usage of Kylin and AWS.

1. Deploying Apache Kylin on AWS
And designing a task scheduler for it
Chase Zhang
Strikingly

2. Outline
Introduction
Strikingly
Analytics Service of Strikingly
Deploy Apache Kylin on AWS
Overview
Containerizing Kylin
Maintenance
Scheduler for Kylin System
Designing Goals
Basic Idea & Implementation
Tasks, Executors and Services
Concurrency and Fault Tolerance
Maintenance and Monitoring
Conclusion

3. Introduction
Strikingly
https://strikingly.com
https://sxl.com

4. Introduction
Strikingly
At Strikingly, we are devoted to provide convenient and one stop website building solution
to our customers.

5. Introduction
Analytics Service of Strikingly
The version 0 of our analytics service is Google Analytics
Strikingly
Google
Analytics
User Pages
User
Register / Get Track IDSet Track ID
Generate User's website
Collect Page Views Data
Serve User Query
Figure: Google Analytics

6. Introduction
Analytics Service of Strikingly
The version 1 of our analytics service is through Keen IO, a 3rd party service
Strikingly
User Pages
User
View Analytics
Generate User's website
Keen.IO
Serve User Query
Collect Page Views Data
Figure: Keen IO

7. Introduction
Analytics Service of Strikingly
The version 2 of our analytics services is combining Keen IO and Apache Kylin
Strikingly
User Pages
User
View Analytics
Generate User's website
Keen.IO
Collect Page Views Data
Apache
Kylin
Serve User Query
Figure: Keen IO + Apache Kylin

8. Deploy Apache Kylin on AWS
Overview
EMR ECS
7070
33345
33345Kylin
(query)
Kylin
(query)
Kylin
(query)
Kylin
(job)
Application
Load
Balancer
Hadoop Hive HBase
Hive HBase
S3
33347
Target
Group
80
Hadoop
Hadoop Hive
YARN
HBase
Query
Requests
Keen.IO
Figure: Deploy Apache Kylin on AWS

9. Deploy Apache Kylin on AWS
Containerizing Kylin
Hive HBase
HDFS
Apache Kylin
MapReduce / Spark
Figure: Apache Kylin is “Stateless”

10. Deploy Apache Kylin on AWS
Containerizing Kylin
Problem
We’d like to
▶ Deploy Kylin on multiple regions
▶ Customize behaviors with environment variables
▶ Build a single docker image and run everywhere

11. Deploy Apache Kylin on AWS
Containerizing Kylin
Hive HBase Hadoop YARN Kylin
Configuration Files Templates
Configuration Files
Substitute variables
Start running
Figure: Launching Kylin with customized script

12. Deploy Apache Kylin on AWS
Maintenance
Problem
Two problems while maintaining this system:
▶ Auto scale and dynamic ports
▶ Clean-up and back-up

13. Deploy Apache Kylin on AWS
Maintenance
ECS
33345
33345
xxxxx
Kylin
(query)
Kylin
(query)
Kylin
(query)
Application
Load
Balancer
33347
Target
Group
80
Query
Requests
Kylin
(query)7070Kylin
(job)
Figure: Auto Scale and Dynamic Listening Ports

14. Deploy Apache Kylin on AWS
Maintenance
./bin/metastore.sh backup
./bin/metastore.sh restore
./bin/metastore.sh clean
./bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob
Figure: Clean-up and back-up tools

15. Deploy Apache Kylin on AWS
Maintenance
Solution
A customized task scheduler.

16. Scheduler for Kylin System
Designing Goals
▶ Customizing task scheduling
▶ Making system robust and fault tolerant
▶ Solving both previously mentioned maintenance problems

17. Scheduler for Kylin System
Basic Idea & Overall Design
The Systemd (Anti-UNIX) philosophy
▶ Scheduler works as a central service
▶ Other components work as RPC services

18. Scheduler for Kylin System
Basic Idea & Overal Design
Scheduler
Target Group
Kylin
(query)
Kylin
(query)
Kylin
(query)
Kylin
(query)
Kylin
(job)
HBase Hive
DynamoDB S3
Keen.IO
Kylin
(query)

19. Scheduler for Kylin System
Basic Idea & Overall Design
Implementation details:
▶ Applying FP9
and Actor Model10
ideas
▶ Implemented with Scala11
and Akka12
▶ Interact with Hadoop components through Java libraries
9
https://en.wikipedia.org/wiki/Functional_programming
10
https://en.wikipedia.org/wiki/Actor_model
11
http://scala-lang.org/
12
https://akka.io/

20. Scheduler for Kylin System
Basic Idea & Overall Design
Control Actor
Consistent
Hashing Router
Task Actor
Executor
Scheduler
1 2
1
22
3
3
3
1
2
3
1
Control Message
Task Message
Service
Figure: Scheduler’s Actor System

21. Scheduler for Kylin System
Tasks, Executors and Services
▶ Task = immutable message
▶ Task has a type for executor
▶ Executor call services to work
▶ Task categories: planning tasks, working tasks, maintaining tasks

22. Scheduler for Kylin System
Tasks, Executors and Services
PlanDataRefresh
PlanCubeMaintenance
HiveTableRefresh
KylinCubeBuild
KylinCubeRefresh
KylinCubeMerge
Hive Service
Kylin Service
Hourly
Daily
Need import new data?
Need build a new segment?
Need refresh old segments?
Need fill holes between segments?
Need merge segments?
Need fill holes in hive table?
Hive table has been refreshed, refresh segment
Planning Tasks Working Tasks Services
Message Storage
Service
Figure: Planning Tasks and Working Tasks

23. Scheduler for Kylin System
Tasks, Executors and Services
KylinMetadataBackup
KylinMetadataCleanup
KylinMetadataRestore
KylinHBaseTableCleanup
HBase Service
Kylin Service
AWS Service
S3
Apache Kylin
KYLIN_XWFQ12
kylin_metadata
kylin-metadata-backups
Update Cache
Get Cube Info
Delete Table
Read MetadataDelete Row
Write ZIP File
Read ZIP File
Write Table
Get Cube Info
Figure: Maintaining Tasks

24. Scheduler for Kylin System
Concurrency and Fault Tolerance
Problem
We’d like to execute tasks in order
▶ Maintaining tasks run exclusively
▶ Tasks of the same cube run execlusively

25. Scheduler for Kylin System
Concurrency and Fault Tolerance
Solution
Two manners to solve this problem:
▶ ReadWriteLock
▶ ConsistentHashingRouter

26. Scheduler for Kylin System
Concurrency and Fault Tolerance
Problem
We’d like to be fault tolerant:
1. Recovering from failures
2. Filling missed segment gaps
3. Recording history

27. Scheduler for Kylin System
Concurrency and Fault Tolerance
Solution
We’re taking multiple manners to solve this problem:
1. Assigning each task with a Unique ID
2. Persisting task message with progress to DynamoDB
3. Implementing planning and working tasks carefully to be issue aware

28. Scheduler for Kylin System
Concurrency and Fault Tolerance
ControlActor TaskActor
Executor
Consistent
HashingRouterinit running finish
error
DynamoDB DynamoDB DynamoDB
TaskMessage
Acquire Lock Release Lock
Figure: Concurrency and Message Persistent

29. Scheduler for Kylin System
Maintenance and Monitoring
Problem
We still have two trival problems to solve:
▶ Manually performing actions
▶ Task monitoring and error notification

30. Scheduler for Kylin System
Maintenance and Monitoring
How to design the user interface of scheduler?

31. Scheduler for Kylin System
Maintenance and Monitoring
Introducing scheduler slack bot...

32. Scheduler for Kylin System
Maintenance and Monitoring
Event Bus
Control Actor
Consistent
Hashing Router Task Actor Executor Service
SlackBot Actor
User Command
Figure: Scheduler Slack Bot

33. Scheduler for Kylin System
Maintenance and Monitoring
Figure: List task status

34. Scheduler for Kylin System
Maintenance and Monitoring
Figure: List Kylin Job Progress

35. Conclusion
▶ With Apache Kylin, we’re providing a sub-second web analytics service
▶ With little effort, we managed to deploy Apache Kylin with docker container
▶ With the scheduler, we deployed the system on AWS without losses of features
▶ We’ve made the system concurrency safe and robust

36. Conclusion
Version 3?
But wait, we still have a problem, don’t we?

37. Conclusion
Version 3?
User
Keen.IOS3
North America
S3
Tokyo, Japan
S3
Beijing, China
User
5 minutes10 minutes20 minutes
Page Views
Figure: Data Transfer Delay of Keen IO

38. Conclusion
Version 3?
User
S3
Tokyo, Japan
Application
Load Balancer
S3
Beijing, China
User
5 minutes
Application
Load Balancer
5 minutes
Page Views Page Views
Figure: Collecting Data with ALB?

39. Thank you!
BTW, we’re still hiring Data Platform
Engineer:
1. Writing Scala
2. Working on AWS
3. Working with Apache Kylin
4. Working on our “Project Manhattan”