利用 AWS Step Functions 建構穩健的資料處理流程 Level: 200, 中文演講 講師: Kwun-hok Chan, Solutions Architect, AWS & CoCo Lin, Engineer, TrendMicro & Bruce Wy Lin, Senior Engineer, TrendMicro 當企業開始普遍使用 Machine Learning/AI 相關技術時，隨之而來的，便是如何管理眾多的資料處理流程。 面對大量且多樣的資料，確保資料正確與一致，便成為一件複雜但又不得不做好的事情。 本技術分享將介紹趨勢科技如何利用 AWS Step Functions 服務，大幅簡化資料處理架構， 在短時間內建構出繁雜的資料驗證流程，並介紹如何利用 AWS Step Functions 實作出實務上必須 面對的 timeout、retry 與 error handling 等機制，以確保資料處理流程能夠穩健運作。 聯繫銷售: https://aws.amazon.com/tw/contact-us/ 與銷售線上聊天: https://pages.awscloud.com/tw-hkt-sales-chat.html

1. S U M M I T
Taipei

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Step Functions
Kwunhok Chan
Solutions Architect
Amazon Web Services
S e s s i o n I D
CoCo Lin
Engineer
Trend Micro
Bruce WY Lin
Senior Engineer
Trend Micro

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
No servers to provision
or manage
Scales with usage
Never pay for idle Availability and
fault-tolerance built in
Serverless means…

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
SERVICES (ANYTHING)
Changes in
data state
Requests to
endpoints
Changes in
resource state
EVENT SOURCE FUNCTION
Node.js
Python
Java
C#
Go
Ruby
and more…
Serverless applications

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
DBMSQueue
Modern App

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Modern App

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Turning functions into apps
“I want to select functions based on data”
“I want to retry functions”
“I want try/catch/finally”
“I have code that runs for hours”
“I want to run functions in parallel”
“I want to sequence functions”

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Example Use Case
Suppose I want to
book my vacation to
the Grand Canyon,
…but unwind my reservations if I can’t rent a car.

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordination by method call

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordination by function chaining

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordination by database

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordination by queues

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS
Step Functions

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Benefits of AWS Step Functions
Manages the logic of your
application. This removes
extra code that may be
repeated in your
microservices and functions.
Easy to connect and
coordinate
distributed components and
microservices to quickly
create apps
AWS Step Functions
manages state, checkpoints
and restarts to make sure
your application executes in
order and as expected.
Build and update
Apps Quickly
Write Less Code Improve
Resiliency

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Run each step in sequence
Vendor A
Vendor B
Vendor C

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
…and unwind if my plans fail at any point
Vendor A
Vendor B
Vendor C

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordination must-haves
• Scales out
• Doesn’t lose state
• Deals with errors/timeouts
• Easy to build & operate
• Auditable

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Application lifecycle in AWS Step Functions
Visualize in the
Console
Define in JSON Monitor
Executions

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Seven state types
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data across tasks
Wait Delay for a specified time
Fail Stops an execution and marks it as a failure
Succeed Stops an execution successfully
Pass Passes its input to its output

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Task states for any compute
Long poll
Traditional server
Request
Worker requests tasks
from Step Functions
Step Functions
invokes the Task

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Build visual workflows from state types
Task
Choice
Fail
Parallel

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Common Use Cases
• Build modern applications
• Data processing
• Automate tasks
• Break-up monoliths
• Automate Machine Learning (ML) jobs

23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Big Data AI/ML
Data

25. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Raw Data
Behavior
Features
Static
Features
Verify Data Analyze

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
[ "static_features_1.log",
"static_features_2.log" ]
{"schema": "static_features",
"result": "passed"}

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
{
"static_features": [
"static_features_1.log",
"static_features_2.log"],
…
…
…
”behavior_features": [
"behavior_features_1.log",
"behavior_features_2.log"],
…
…
…
}
Get value from key “static_features” :
["static_features_1.log",
"static_features_2.log"]
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
Get value from key “static_features” :
["static_features_1.log",
"static_features_2.log"]
Verification
on
EMR
Get value from key “static_features” :
["static_features_1.log",
"static_features_2.log"]
Verification
on
EMR
Get value from key “behavior_features” :
[" behavior_features _1.log",
" behavior_features _2.log"]
Verification
on
EMR

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
["static_features_1.log",
"static_features_2.log"]
Parallel
InputPath=$.static_features
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
{
"static_features": [
"static_features_1.log",
"static_features_2.log"],
…
…
…
”behavior_features": [
"behavior_features_1.log",
"behavior_features_2.log"],
…
…
…
}

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
["static_features_1.log",
"static_features_2.log"]
Parallel
InputPath=$.static_features
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
["behavior_features _1.log",
"behavior_features _2.log"]
InputPath=$.static_features

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
["static_features_1.log",
"static_features_2.log"]
Parallel
InputPath=$.static_features
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
["behavior_features _1.log",
"behavior_features _2.log"]
InputPath=$.static_features
Fail

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
["static_features_1.log",
"static_features_2.log"]
Parallel
InputPath=$.static_features
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
["behavior_features _1.log",
"behavior_features _2.log"]
InputPath=$.static_features
Fail

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
["static_features_1.log",
"static_features_2.log"]
Parallel
InputPath=$.static_features
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
Succeed
Choice
Fail
Verification
on
EMR
Task
Wait
["behavior_features _1.log",
"behavior_features _2.log"]
InputPath=$.static_features
Fail

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Succeed
Choice
Pass
Task
Wait

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Succeed
Choice
Pass
Task
Wait

36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Infinite Loop
• Your bill may be infinite, too
SucceedPass

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Dead and… End
• Often happened while waiting
• Do nothing
• Get nothing
SucceedPass

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Only One Timeout
• No timeout for sub-
workflow
SucceedPass SucceedPass
Parallel
8 mins 24 mins

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Risk Summary Run It
Infinite Loop
Add
Timeout
Dead and… End
Adjust
Timeout
Only
One
Timeout

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Simplified Job Status Poller
Poller
Submitter
Error
Handler
Succeed
SucceedPass

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
If emr_status == ‘COMPLETED’:
pass
elif emr_status in [‘PENDING’, ‘RUNNING’]:
raise AsyncTaskUnfinished(‘go to retry’)
"Poller":{
"Type": "Task",
"Next": "Succeed",
"ResultPath": "$.output.poller",
"Retry":[
{
"ErrorEquals": ["AsyncTaskUnfinished"],
"IntervalSeconds": 10,
"MaxAttempts": 3,
}
]
}

49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Poller
Submitter
Error
Handler
Succeed
"Poller":{
"Type": "Task",
"Next": "Succeed",
"ResultPath": "$.output.poller",
"Retry":[
{
"ErrorEquals": ["AsyncTaskUnfinished"],
"IntervalSeconds": 10,
"MaxAttempts": 3,
"BackoffRate": 2.0
}
]
}

50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Custom Errors Save Life
"Poller":{
"Type": "Task",
"Next": "Succeed",
"ResultPath": "$.output.poller",
"Retry":[
{
"ErrorEquals": ["AsyncTaskUnfinished"],
"IntervalSeconds": 10,
"MaxAttempts": 3,
"BackoffRate": 2.0
}
],
"Catch":[
{
"ErrorEquals": [" AsyncTaskFailed"],
"ResultPath": "$.error.poller",
"Next": "ErrorHandler"
}
]
},
"ErrorHandler": {
"Type": "Task",
"ResultPath": "$.error",
"End": true
}
if emr_status in [‘CANCELLED’, ‘FAILED’,
‘INTERRUPTED’]:
raise AsyncTaskFailed(‘go to error handler ’)

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
"PollerA":{
"Type": "Task",
"Next": "Succeed",
"ResultPath": "$.output.poller",
"Retry":[
{
"ErrorEquals": ["AsyncTaskUnfinished"],
"IntervalSeconds": 10,
"MaxAttempts": 3,
}
],
},
"PollerB":{
"Type": "Task",
"Next": "Succeed",
"ResultPath": "$.output.poller",
"Retry":[
{
"ErrorEquals": ["AsyncTaskUnfinished"],
"IntervalSeconds": 5,
"MaxAttempts": 100,
}
],
},
Parallel
Start
PollerA
Submitter
Error
Handler
Succeed
30s
PollerB
Submitter
Error
Handler
Succeed
500s

54. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Step Functions

56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
3 Problems
Infinite Loop
Dead and… End
Only One Timeout
3 Solutions
Retry Has Max
Custom Errors Save Life
Retry Makes Timeout Anywhere

57. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Better Step Functions
Step Functions Better

58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kwunhok Chan
Solutions Architect
Amazon Web
Services
CoCo Lin
Engineer
Trend Micro
Bruce WY Lin
Senior Engineer
Trend Micro

59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.