More Related Content
Similar to Digdag Updates 2020 July (20)
Digdag Updates 2020 July
- 1. © 2020 Arm Limited
Digdag updates
Embulk & Digdag Online Meetup 2020
1
- 2. © 2019 Arm Limited
About me
• You Yamagata (github: yoyama)
• Joined Arm Treasure Data in Jan. 2019
• Backend team
• TD Workflow/Digdag
2
- 3. © 2019 Arm Limited
Agenda
• v0.9 recent updates
• Next major version
• Graaljs migration in TD
- 4. © 2019 Arm Limited
About TD and Digdag
• TD is providing workflow in TD service
https://tddocs.atlassian.net/wiki/spaces/PD/pages/1083994/Understanding+Workflows
• For both customers and internal use
• TD workflow is based on Digdag
• Not using v0.9, but using another version (in public)
• Customize Digdag
– Private code is small (e.g. authentication, audit)
• Running many workflows in clusters with multi tenant
Digdag is an important component for TD workflow!
TD workflow is the largest use case of Digdag
- 6. © 2019 Arm Limited
Feature highlight
1. http_call> 0.9.37
2.
Authenticator SPI & plugin /
Basic authentication
0.9.39
3. GCS support as storage 0.9.41
4. require> enhancements 0.9.42
6
- 7. © 2019 Arm Limited
http_call>
• 0.9.37
• New operator
https://docs.digdag.io/operators/http_call.html
• Similar to call>
• call> ... Insert another workflow in the project
• http_call> ... Insert another workflow downloaded
The response of http must be either YAML or JSON
• Need correct content-type or content_type_override param
• Use case
• Dynamically generate workflow with complicated conditions
7
- 8. © 2019 Arm Limited
Authenticator SPI & plugin / Basic authentication
• 0.9.39
• New SPI and plugin configuration for developer
• io.digdag.spi.Authenticator
• server.authenticator-class
• Introduce basic authentication plugin
8
- 9. © 2019 Arm Limited
Setup basic authentication
9
server.properties
server.authenticator-class = io.digdag.standards.auth.basic.BasicAuthenticator
basicauth.username = user1
basicauth.password = testtest
> digdag projects --basic-auth user1:testtest
2020-07-02 18:48:53 +0900: Digdag v0.9.42
Projects
name: prj1
id: 1
> cat ~/.config/digdag/config
client.http.headers.authorization = Basic dXNlcjE6dGVzdHRlc3Q
> digdag -c ~/.config/digdag/config projects
- 10. © 2019 Arm Limited
Develop authenticator plugin
10
• digdag-basic-auth-mt
https://github.com/yoyama/digdag-basic-auth-mt
• Basic authentication with
• multi-user
• multi-tenant (site_id)
server.properties
server.authenticator-class = io.github.yoyama.digdag.bauth.BasicAuthMTAuthenticator
system-plugin.repositories = https://dl.bintray.com/yoyama/maven
system-plugin.dependencies = io.github.yoyama:digdag-basic-auth-mt_2.12:0.1.0
plugins.basic_auth_mt.passwd_file=<path_to_passwrwd_file>
plugins.basic_auth_mt.user_map_file=<path_to_user_map_fiel>
passwd_file (htpasswd)
user2:$apr1$HtOrA/2I$HXTTf2p4MQqbVV/Alq4ci.
user1:$apr1$xjBNYkXH$HJ7pXsaX9MXAyxfgl6uLY/
user_map_file (user:site_id:admin)
user1:0:true
user2:22:false
- 11. © 2019 Arm Limited
GCS support as storage
• 0.9.39
• digdag-storage-gcs
• Similar to digdag-storage-s3
• Archive and log-server
11
- 12. © 2019 Arm Limited
require> enhancement
• 0.9.42 latest
• Back port from v0_10
• project_name parameter
• rerun_on parameter
• Fix an issues on ‘digdag retry’
https://github.com/treasure-data/digdag/issues/712
12
- 13. © 2019 Arm Limited
rerun_on
• Both Parent1 and parent2 call child1
• All run is in same session time
• Parent2 run firstly and kick child1
• Parent1 will start after parent2 finish.
parent1.dig
+p1:
require>: child1
child1.dig
+c1:
...
parent2.dig
+p2:
require>: child1
parent2
child1
parent1
?
+p2:
+p1:
- 14. © 2019 Arm Limited
rerun_on: none
• rerun_on: none (default)
• child1 kicked by parent2: success
parent1.dig
+p1:
require>: child1
rerun_on: none
child1.dig
+c1:
...
parent2.dig
+p2:
require>: child1
child1
success
child1
skip
• parent1 will not kick child1
• +p1 will succeed
parent2
parent1
+p2:
+p1:
- 15. © 2019 Arm Limited
rerun_on: none
• rerun_on: none (default)
• child1 kicked by parent2: fail
parent1.dig
+p1:
require>: child1
rerun_on: none
child1.dig
+c1:
...
parent2.dig
+p2:
require>: child1
child1
fail
child1
skip
• parent1 will not kick child1
• +p1 will fail
parent2
parent1
+p2:
+p1:
- 16. © 2019 Arm Limited
rerun_on: failed
• rerun_on: failed
• child1 kicked by parent2: success
parent1.dig
+p1:
require>: child1
rerun_on: none
child1.dig
+c1:
...
parent2.dig
+p2:
require>: child1
child1
success
child1
skip
• parent1 will not kick child1
• +p1 will succeed
parent2
parent1
+p2:
+p1:
- 17. © 2019 Arm Limited
rerun_on: failed
• rerun_on: failed
• child1 kicked by parent2: fail
parent1.dig
+p1:
require>: child1
rerun_on: none
child1.dig
+c1:
...
parent2.dig
+p2:
require>: child1
child1
fail
child1
kicked
• parent1 will kick child1
parent2
parent1
+p2:
+p1:
- 18. © 2019 Arm Limited
rerun_on: all
• rerun_on: all
• child1 kicked by parent2: success
parent1.dig
+p1:
require>: child1
rerun_on: none
child1.dig
+c1:
...
parent2.dig
+p2:
require>: child1
child1
success
child1
kicked
• parent1 will kick child1
parent2
parent1
+p2:
+p1:
- 19. © 2019 Arm Limited
rerun_on: all
• rerun_on: all
• child1 kicked by parent2: fail
parent1.dig
+p1:
require>: child1
rerun_on: none
child1.dig
+c1:
...
parent2.dig
+p2:
require>: child1
child1
fail
child1
kicked
• parent1 will kick child1
parent2
parent1
+p2:
+p1:
- 20. © 2019 Arm Limited
require> enhancement
rerun_on control kick of child when child attempts exist
• rerun_on: none (default)
• Never kick. Same as previous version
• rerun_on: failed
• Kick if the attempt failed
• rerun_on: all
• Kick always
20
- 22. © 2019 Arm Limited
Current status
22
• Under development (v0.10.0)
• Based on TD release branch
• v0_10
• Include many modification, features
• Release schedule is not clear.
• 0.9 is still mainline
• If you want to try ...
• Build v0_10 branch
• Download snapshot https://bintray.com/digdag/maven-snapshots/...
• Just only for evaluation
- 23. © 2019 Arm Limited
Feature highlight
23
• CommandExecutor / ECS,K8 support
• AccessController SPI
• Java11
• Graaljs
- 24. © 2019 Arm Limited
CommandExecutor
24
What is CommandExecutor?
• “Command executor (io.digdag.spi.CommandExecutor) is a plugin interface that is
used to execute a command in a sandbox environment.”
http://docs.digdag.io/internal.html#command-executor
• Running tasks in Docker, container services
• sh>, py>, rb> operators support CommandExecutor
• 0.9 supports
• DockerCommandExecutor
• SimpleCommandExectuor
Default is docker
Fallback if no docker setting
- 25. © 2019 Arm Limited
CommandExecutor
25
New in 0.10
• SPI has been modified (support container service)
• Support
• ECSCommandExecutor
• KubernetesCommandExecutor
• Under development
- 26. © 2019 Arm Limited
Java11
26
• Experimental
• Fixed some blockers
• Now you can build and run on Java11
• Java8 is main platform in Digdag
• Java8 is still alive
• Both support is hard for us
- 27. © 2019 Arm Limited
Graaljs
27
• Support graaljs as replacement of nashorn
• Nashorn is default in Java8
• Graaljs is default in Java11
• Configuration
• eval.js-engine-type : "nashorn" or "graal"
• eval.extended-syntax : true or false
– only support in graal
• syntax extension
• Support nested ‘{...}’
• But ...
_export:
v1: [1,2,3]
+t1:
echo>: "${v1.map(function(item){ return item*5 } )}"
- 29. © 2019 Arm Limited
Background
29
• A plan to move on Java11 in TD workflow
• Nashorn as JavaScript engine in workflow
definition “ConfigEvalEngine”
• Nashorn is deprecated in Java11 blocker!
• Nashorn also causes performance degradation
• We need to replace Nashorn to another
Graaljs
- 30. © 2019 Arm Limited
Graaljs
30
• “A high performance implementation of the JavaScript programming language. Built
on the GraalVM by Oracle Labs”
https://github.com/graalvm/graaljs
• Graaljs is available in JDK
- 31. © 2019 Arm Limited
Purpose / Requirements
31
• Improve performance of ConfigEvalEngine
• Keep compatibility between nashorn and graaljs
• Workflow behavior must be same as much as possible
• Stability
- 32. © 2019 Arm Limited
Procedure
32
1. A PR (by sada)
2. Pre-investigation performance/compatibility
3. Some fixes for improvements
4. Run in TD production for compatibility test
5. Fix issues
6. Release
- 33. © 2019 Arm Limited
Pre-investigation and improvement
33
• Extract graaljs and nashorn logic from Digdag and evaluated
• Point 1 … performance
• Point 2 … compatibility
- 34. © 2019 Arm Limited
Pre-investigation and improvement
34
• Performance
• Evaluated by small test code with JMH
• Graaljs is very faster than nashorn in both
Java8 and Java11
• Compatibility
• Optimize graaljs options and improve
compatibility
• Graaljs has “js.nashorn-compat” but not
enough
- 35. © 2019 Arm Limited
Conclusion on pre-investigation
35
• Graaljs performance is enough for Digdag
• Compatibility is basically good
- 36. © 2019 Arm Limited
Run compatibility test in production
36
• Used special mode
• Both nashorn and graaljs evaluate same text and compare the result
• Stop using syntax extension
• Found some issues only in production
• Iterate fix and test
• Finally fixed all issues
“echo>: ${moment(....)..”
ConfigEvalEngine
nashorn
graaljs
compare
discard
return value
logging
- 37. © 2019 Arm Limited
Results
37
• We had already deployed Graaljs/w Java11
• No inquiry on workflow compatibility
• Performance improved dramatically as expected
• CPU usage declined
- 38. © 2019 Arm Limited
Conclusion
38
• Graaljs is very fit to Digdag
• Almost all efforts and works in TD has been contributed to Digdag (v0_10)
• Users don’t care compatibility of JS on workflow definition with graaljs
