Download as PDF, PPTX
![© 2019 Arm Limited
Graaljs
27
• Support graaljs as replacement of nashorn
• Nashorn is default in Java8
• Graaljs is default in Java11
• Configuration
• eval.js-engine-type : "nashorn" or "graal"
• eval.extended-syntax : true or false
– only support in graal
• syntax extension
• Support nested ‘{...}’
• But ...
_export:
v1: [1,2,3]
+t1:
echo>: "${v1.map(function(item){ return item*5 } )}"](https://image.slidesharecdn.com/digdagupdateatmeetup2020july-200710071328/85/Digdag-Updates-2020-July-27-320.jpg)
![[212] large scale backend service develpment](https://cdn.slidesharecdn.com/ss_thumbnails/221largescalebackendservicedevelpment-150915001346-lva1-app6892-thumbnail.jpg?width=640&height=640&fit=bounds)
![[245] presto 내부구조 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/245presto-150915054242-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
Digdag Updates 2020 July
- 1. © 2020 ArmLimited Digdag updates Embulk & Digdag Online Meetup 2020 1
- 2. © 2019 ArmLimited About me • You Yamagata (github: yoyama) • Joined Arm Treasure Data in Jan. 2019 • Backend team • TD Workflow/Digdag 2
- 3. © 2019 ArmLimited Agenda • v0.9 recent updates • Next major version • Graaljs migration in TD
- 4. © 2019 ArmLimited About TD and Digdag • TD is providing workflow in TD service https://tddocs.atlassian.net/wiki/spaces/PD/pages/1083994/Understanding+Workflows • For both customers and internal use • TD workflow is based on Digdag • Not using v0.9, but using another version (in public) • Customize Digdag – Private code is small (e.g. authentication, audit) • Running many workflows in clusters with multi tenant Digdag is an important component for TD workflow! TD workflow is the largest use case of Digdag
- 5. © 2020 ArmLimited v0.9 recent updates 5
- 6. © 2019 ArmLimited Feature highlight 1. http_call> 0.9.37 2. Authenticator SPI & plugin / Basic authentication 0.9.39 3. GCS support as storage 0.9.41 4. require> enhancements 0.9.42 6
- 7. © 2019 ArmLimited http_call> • 0.9.37 • New operator https://docs.digdag.io/operators/http_call.html • Similar to call> • call> ... Insert another workflow in the project • http_call> ... Insert another workflow downloaded The response of http must be either YAML or JSON • Need correct content-type or content_type_override param • Use case • Dynamically generate workflow with complicated conditions 7
- 8. © 2019 ArmLimited Authenticator SPI & plugin / Basic authentication • 0.9.39 • New SPI and plugin configuration for developer • io.digdag.spi.Authenticator • server.authenticator-class • Introduce basic authentication plugin 8
- 9. © 2019 ArmLimited Setup basic authentication 9 server.properties server.authenticator-class = io.digdag.standards.auth.basic.BasicAuthenticator basicauth.username = user1 basicauth.password = testtest > digdag projects --basic-auth user1:testtest 2020-07-02 18:48:53 +0900: Digdag v0.9.42 Projects name: prj1 id: 1 > cat ~/.config/digdag/config client.http.headers.authorization = Basic dXNlcjE6dGVzdHRlc3Q > digdag -c ~/.config/digdag/config projects
- 10. © 2019 ArmLimited Develop authenticator plugin 10 • digdag-basic-auth-mt https://github.com/yoyama/digdag-basic-auth-mt • Basic authentication with • multi-user • multi-tenant (site_id) server.properties server.authenticator-class = io.github.yoyama.digdag.bauth.BasicAuthMTAuthenticator system-plugin.repositories = https://dl.bintray.com/yoyama/maven system-plugin.dependencies = io.github.yoyama:digdag-basic-auth-mt_2.12:0.1.0 plugins.basic_auth_mt.passwd_file=<path_to_passwrwd_file> plugins.basic_auth_mt.user_map_file=<path_to_user_map_fiel> passwd_file (htpasswd) user2:$apr1$HtOrA/2I$HXTTf2p4MQqbVV/Alq4ci. user1:$apr1$xjBNYkXH$HJ7pXsaX9MXAyxfgl6uLY/ user_map_file (user:site_id:admin) user1:0:true user2:22:false
- 11. © 2019 ArmLimited GCS support as storage • 0.9.39 • digdag-storage-gcs • Similar to digdag-storage-s3 • Archive and log-server 11
- 12. © 2019 ArmLimited require> enhancement • 0.9.42 latest • Back port from v0_10 • project_name parameter • rerun_on parameter • Fix an issues on ‘digdag retry’ https://github.com/treasure-data/digdag/issues/712 12
- 13. © 2019 ArmLimited rerun_on • Both Parent1 and parent2 call child1 • All run is in same session time • Parent2 run firstly and kick child1 • Parent1 will start after parent2 finish. parent1.dig +p1: require>: child1 child1.dig +c1: ... parent2.dig +p2: require>: child1 parent2 child1 parent1 ? +p2: +p1:
- 14. © 2019 ArmLimited rerun_on: none • rerun_on: none (default) • child1 kicked by parent2: success parent1.dig +p1: require>: child1 rerun_on: none child1.dig +c1: ... parent2.dig +p2: require>: child1 child1 success child1 skip • parent1 will not kick child1 • +p1 will succeed parent2 parent1 +p2: +p1:
- 15. © 2019 ArmLimited rerun_on: none • rerun_on: none (default) • child1 kicked by parent2: fail parent1.dig +p1: require>: child1 rerun_on: none child1.dig +c1: ... parent2.dig +p2: require>: child1 child1 fail child1 skip • parent1 will not kick child1 • +p1 will fail parent2 parent1 +p2: +p1:
- 16. © 2019 ArmLimited rerun_on: failed • rerun_on: failed • child1 kicked by parent2: success parent1.dig +p1: require>: child1 rerun_on: none child1.dig +c1: ... parent2.dig +p2: require>: child1 child1 success child1 skip • parent1 will not kick child1 • +p1 will succeed parent2 parent1 +p2: +p1:
- 17. © 2019 ArmLimited rerun_on: failed • rerun_on: failed • child1 kicked by parent2: fail parent1.dig +p1: require>: child1 rerun_on: none child1.dig +c1: ... parent2.dig +p2: require>: child1 child1 fail child1 kicked • parent1 will kick child1 parent2 parent1 +p2: +p1:
- 18. © 2019 ArmLimited rerun_on: all • rerun_on: all • child1 kicked by parent2: success parent1.dig +p1: require>: child1 rerun_on: none child1.dig +c1: ... parent2.dig +p2: require>: child1 child1 success child1 kicked • parent1 will kick child1 parent2 parent1 +p2: +p1:
- 19. © 2019 ArmLimited rerun_on: all • rerun_on: all • child1 kicked by parent2: fail parent1.dig +p1: require>: child1 rerun_on: none child1.dig +c1: ... parent2.dig +p2: require>: child1 child1 fail child1 kicked • parent1 will kick child1 parent2 parent1 +p2: +p1:
- 20. © 2019 ArmLimited require> enhancement rerun_on control kick of child when child attempts exist • rerun_on: none (default) • Never kick. Same as previous version • rerun_on: failed • Kick if the attempt failed • rerun_on: all • Kick always 20
- 21. © 2020 ArmLimited Next major version 21
- 22. © 2019 ArmLimited Current status 22 • Under development (v0.10.0) • Based on TD release branch • v0_10 • Include many modification, features • Release schedule is not clear. • 0.9 is still mainline • If you want to try ... • Build v0_10 branch • Download snapshot https://bintray.com/digdag/maven-snapshots/... • Just only for evaluation
- 23. © 2019 ArmLimited Feature highlight 23 • CommandExecutor / ECS,K8 support • AccessController SPI • Java11 • Graaljs
- 24. © 2019 ArmLimited CommandExecutor 24 What is CommandExecutor? • “Command executor (io.digdag.spi.CommandExecutor) is a plugin interface that is used to execute a command in a sandbox environment.” http://docs.digdag.io/internal.html#command-executor • Running tasks in Docker, container services • sh>, py>, rb> operators support CommandExecutor • 0.9 supports • DockerCommandExecutor • SimpleCommandExectuor Default is docker Fallback if no docker setting
- 25. © 2019 ArmLimited CommandExecutor 25 New in 0.10 • SPI has been modified (support container service) • Support • ECSCommandExecutor • KubernetesCommandExecutor • Under development
- 26. © 2019 ArmLimited Java11 26 • Experimental • Fixed some blockers • Now you can build and run on Java11 • Java8 is main platform in Digdag • Java8 is still alive • Both support is hard for us
- 27. © 2019 ArmLimited Graaljs 27 • Support graaljs as replacement of nashorn • Nashorn is default in Java8 • Graaljs is default in Java11 • Configuration • eval.js-engine-type : "nashorn" or "graal" • eval.extended-syntax : true or false – only support in graal • syntax extension • Support nested ‘{...}’ • But ... _export: v1: [1,2,3] +t1: echo>: "${v1.map(function(item){ return item*5 } )}"
- 28. © 2020 ArmLimited Graaljs migration in TD 28
- 29. © 2019 ArmLimited Background 29 • A plan to move on Java11 in TD workflow • Nashorn as JavaScript engine in workflow definition “ConfigEvalEngine” • Nashorn is deprecated in Java11 blocker! • Nashorn also causes performance degradation • We need to replace Nashorn to another Graaljs
- 30. © 2019 ArmLimited Graaljs 30 • “A high performance implementation of the JavaScript programming language. Built on the GraalVM by Oracle Labs” https://github.com/graalvm/graaljs • Graaljs is available in JDK
- 31. © 2019 ArmLimited Purpose / Requirements 31 • Improve performance of ConfigEvalEngine • Keep compatibility between nashorn and graaljs • Workflow behavior must be same as much as possible • Stability
- 32. © 2019 ArmLimited Procedure 32 1. A PR (by sada) 2. Pre-investigation performance/compatibility 3. Some fixes for improvements 4. Run in TD production for compatibility test 5. Fix issues 6. Release
- 33. © 2019 ArmLimited Pre-investigation and improvement 33 • Extract graaljs and nashorn logic from Digdag and evaluated • Point 1 … performance • Point 2 … compatibility
- 34. © 2019 ArmLimited Pre-investigation and improvement 34 • Performance • Evaluated by small test code with JMH • Graaljs is very faster than nashorn in both Java8 and Java11 • Compatibility • Optimize graaljs options and improve compatibility • Graaljs has “js.nashorn-compat” but not enough
- 35. © 2019 ArmLimited Conclusion on pre-investigation 35 • Graaljs performance is enough for Digdag • Compatibility is basically good
- 36. © 2019 ArmLimited Run compatibility test in production 36 • Used special mode • Both nashorn and graaljs evaluate same text and compare the result • Stop using syntax extension • Found some issues only in production • Iterate fix and test • Finally fixed all issues “echo>: ${moment(....)..” ConfigEvalEngine nashorn graaljs compare discard return value logging
- 37. © 2019 ArmLimited Results 37 • We had already deployed Graaljs/w Java11 • No inquiry on workflow compatibility • Performance improved dramatically as expected • CPU usage declined
- 38. © 2019 ArmLimited Conclusion 38 • Graaljs is very fit to Digdag • Almost all efforts and works in TD has been contributed to Digdag (v0_10) • Users don’t care compatibility of JS on workflow definition with graaljs
- 39.