Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Teamtalk - Cluster Computing Library for Pharo Smalltalk

323 views

Published on

Teamtalk is a simple library to execute Pharo Smalltalk code in a cluster of Pharo Smalltalk instances. MapReduce is also supported.
Based on Vert.x using the VerStix library.
https://github.com/quentinplessis/Teamtalk

Published in: Technology
  • Be the first to comment

Teamtalk - Cluster Computing Library for Pharo Smalltalk

  1. 1. TEAMTALK 第109SMALLTALK勉強会 CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK Quentin Plessis (カンタン プレシ)
  2. 2. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK BACKGROUND ▸ Multi processing is not natively possible in Pharo Smalltalk ▸ To perform resource intensive operations, it is necessary to split the load between several Pharo instances ▸ Efficiently setting up several Pharo instances is troublesome ▸ Establishing communication between several Pharo instances is tricky
  3. 3. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK BACKGROUND ▸ Multi processing is not natively possible in Pharo Smalltalk ▸ To perform resource intensive operations, it is necessary to split the load between several Pharo instances ▸ Efficiently setting up several Pharo instances is troublesome ▸ Establishing communication between several Pharo instances is tricky Docker image
 https://github.com/mumez/pharo-vnc-supervisor VerStix (Vert.x based communication tool)
 https://github.com/mumez/VerStix
 https://www.slideshare.net/umejava/verstix
  4. 4. ▸ Pharo Smalltalk library making it possible to execute tasks in a cluster of Pharos instances. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK TEAMTALK VerStix Teamtalk Teamtalk Server (Vert.x) Producer Pharo A VerStix Teamtalk Worker Pharo B VerStix Teamtalk Worker Pharo C VerStix Teamtalk Worker Pharo D https://github.com/quentinplessis/Teamtalk
  5. 5. ▸ Pharo Smalltalk library making it possible to execute tasks in a cluster of Pharos instances. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK TEAMTALK VerStix Teamtalk Teamtalk Server (Vert.x) Producer Pharo A VerStix Teamtalk Worker Pharo B VerStix Teamtalk Worker Pharo C VerStix Teamtalk Worker Pharo D TASK 1 TASK 2 TASK 3 https://github.com/quentinplessis/Teamtalk
  6. 6. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK PROTOCOL AProducer TASK 1 CB Looking for Workers Available! Available! Available! Execute task TASK 2Execute task TASK 3Execute task Executed TASK 1 RESULT 1 Executed TASK 2 RESULT 2 Executed TASK 3 RESULT 3
  7. 7. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK TASK FLOW AProducer Execution code Input Result Result process code Task [ :input | input + 1 ] 1 [ :result | result inspect ] ‘[ :input | input + 1 ]’ 1 [ :input | input + 1 ] 1 22 [ :input | input + 1 ] 1 2 [ :result | result inspect ] 2
  8. 8. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK REMARKS ▸ Support multiple producers ▸ Workers can be dynamically added / removed ▸ Pharo instances can be on different machines and use different Pharo versions ▸ All communications are broadcasted throughout the cluster ▸ Security issues are present but currently not dealt with
  9. 9. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK GETTING STARTED ▸ Install Teamtalk in two Pharos images A and B ▸ Setup Teamtalk server (docker must be installed and running) ▸ [Pharo Image A] Create a producer ▸ [Pharo Image B] Create a worker ▸ [Pharo Image A] Add a task to execute docker run -p 8080:8080 plequen/teamtalk-server Metacello new baseline: 'Teamtalk'; repository: 'github://quentinplessis/Teamtalk/pharo-repository'; load. producer := TTProducer host: 'localhost' port: 8080. worker := TTWorker host: 'localhost' port: 8080. task := TTTask executionCode: [ :input | input + 1 ] resultProcessCode: [ :result | result inspect. ] input: 1. producer addTask: task.
  10. 10. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK TASK EXAMPLES producer addTask: (TTTask executionCode: [ 1 inspect. 5 seconds wait. nil ]). producer addTask: (TTTask executionCode: [ 2 inspect. 5 seconds wait. nil ]). producer addTask: (TTTask executionCode: [ 3 inspect. 5 seconds wait. nil ]). producer addTask: (TTTask executionCode: [ 4 inspect. 5 seconds wait. nil ]). https://github.com/quentinplessis/Teamtalk start := DateAndTime now. number := 4. i := 0. number timesRepeat: [ task := TTTask executionCode: [ 10 seconds wait. 'OK' inspect. 'OK' ] resultProcessCode: [ :result | i := i + 1. i = number ifTrue: [ (DateAndTime now asUnixTime - start asUnixTime) inspect ]. ]. producer addTask: task. ]. task := TTTask executionCode: [ ZnClient new get: 'https://www.google.com/'. ] resultProcessCode: [ :result | result inspect. ]. producer addTask: task.
  11. 11. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK CLUSTER MANAGEMENT https://github.com/quentinplessis/Teamtalk#cluster-setup ▸ Install Teamtalk in a Pharo image Teamtalk.image ▸ Setup a Teamtalk server on port 8080 ▸ Add consumers, potentially on different machines ▸ Add a producer requires docker and ruby export TEAMTALK_SERVER_PORT=8080 ruby spawner.rb --create-server $TEAMTALK_SERVER_PORT
 ifconfig | grep inet export TEAMTALK_SERVER_IP=..... ruby spawner.rb --create-consumer --pharo-image ./Teamtalk.image --server-host $TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT ruby spawner.rb --create-consumer --pharo-image ./Teamtalk.image --server-host $TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT ruby spawner.rb --create-consumer --pharo-image ./Teamtalk.image --server-host $TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT ruby spawner.rb --create-producer --pharo-image ./Teamtalk.image --server-host $TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT
  12. 12. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK MAP REDUCE TASK SUB TASK SUB TASK SUB TASK SUB TASK SUB TASK SUB RESULT SUB RESULT SUB RESULT SUB RESULT SUB RESULT RESULT MAP REDUCESPLIT
  13. 13. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK MAP REDUCE WITH TEAMTALK ▸ Examples: https://github.com/quentinplessis/Teamtalk#mapreduce mapReduce := (TTMapReduce splitBlockForInput: [ :input :tasksNumber | "split input into sub inputs here" ] mapBlockForSubInput: [ :subInput | "process sub inputs here" "map sub input to sub result" ] reduceBlockWithCallback: [ :results :callback | "reduce sub results here" callback value: results ]) ttClientClass: TTProducer; host: 'localhost'; port: 8080; yourself. mapReduce input: { } tasksNumber: 4 callbackDo: [ :result | result inspect ].
  14. 14. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK IMPROVEMENTS ▸ Add labels to tasks for selective worker selection ▸ Propagate error handling from worker to producer ▸ Improve security ▸ Improve protocol to support additional languages ▸ Add a vote-based task scheduling system to remove the need for producers to manage their tasks ▸ Add mechanism to handle loss of worker during task execution ▸ …
  15. 15. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK TEAMTALK

×