This document discusses an Erlang coding dojo focused on the "Grep" kata using Erlang clusters. It describes joining an Erlang cluster, sending the grep module and files to peers, workers receiving and processing the files in parallel, and aggregating the results. It also discusses using OTP behaviors like gen_server and supervisor for the worker workflow.
The advent of multicore CPUs have facilitated the enhancement of performance in many application domains. But it has shifted the burden of maximizing the usage of this hardware resource to the developers of operating systems and programming languages and ultimately the software engineers who have to develop the applications.
We first look at the concurrency primitives that modern operating systems provides us to work with, namely processes and threads. With Python, a common approach is to use threads which have shared memory. However this can be difficult to do well as it is easy to run into race conditions and other bugs. And Python threads, unlike OS threads, cannot run at the same time due to the infamous GIL, thus limiting us to only one core.
To utilize multiple cores in Python, an effective and better solution is to use multiple processes, each with their own memory space and provide inter-process communication using message passing. We will look at the multiprocessing module and how to use it for this purpose.
Μεταπρογραµµατισµός κώδικα Python σε γλώσσα γραµµικού χρόνου για αυτόµατη επα...ISSEL
Ο όρος επαλήθευση λογικής κατά την εκτέλεση οριοθετεί ένα πεδίο που εκτείνεται από τον έλεγχο λογισµικού για τη συµµόρφωση µε ένα σύνολο προδιαγραφών, έως την εναρµόνιση µε καλές λογικές πρακτικές κατά τη συγγραφή κώδικα. Στο πλαίσιο αυτό, υλοποιήσαµε τη lovpy, µια ϐιβλιοθήκη µεταπρογραµµατισµού για τη γλώσσα Python, που εισάγει σε αυτή τις δυνατότητες της επαλήθευσης λογικής κατά την εκτέλεση. Ο καθορισµός της πρότυπης λογικής γίνεται χρησιµοποιώντας τη διαισθητική γλώσσα έκφρασης προδιαγραφών Gherkin, ενώ η χρήση της ϐιβλιοθήκης δεν απαιτεί καµία αλλαγή στον υπάρχον κώδικα. Για την υλοποίησή της αξιοποιήσαµε µια σειρά εργαλείων της ϑεωρίας γράφων, της ϑεωρίας τυπικών γλωσσών, της χρονικής λογικής καθώς και µοντέλα ϐαθιάς µηχανικής µάθησης, εστιάζοντας περισσότερο στα νευρωνικά δίκτυα γράφων. Θεµελιώσαµε µαθηµατικά ένα νέο είδος γράφου για την αναπαράσταση χρονικών προδιαγραφών και ορίσαµε για αυτόν ένα σύνολο µαθηµατικά τεκµηριωµένων λογικών αλγορίθµων. Στη συνέχεια, αξιοποιήσαµε τις δοµές αυτές προκειµένου να υλοποιήσουµε ένα νέο σύστηµα αυτόµατης απόδειξης ϑεωρη µάτων, το οποίο µας εξασφαλίζει την απόλυτη εγκυρότητα των εντοπισµένων παραβιάσεων. Αξιολογήσαµε πέντε διαφορετικές αποδεικτικές αρχιτεκτονικές, αποτελούµενες από ευριστικούς κανόνες και απλά νευρωνικά µοντέλα, µέχρι µεγάλα νευρωνικά δίκτυα γράφων. Για την εκπαίδευση των νευρωνικών συστηµάτων υλοποιήσαµε ένα µηχανισµό παραγω γής συνθετικών ϑεωρηµάτων, αξιοποιώντας µια σειρά από µαθηµατικές ιδιότητες. Τέλος, χρησιµοποιήσαµε τη lovpy για να εντοπίσουµε σφάλµατα σε δύο δηµοφιλή ϐιβλιοθήκες ανοιχτού κώδικα, την Django και την Keras.
The advent of multicore CPUs have facilitated the enhancement of performance in many application domains. But it has shifted the burden of maximizing the usage of this hardware resource to the developers of operating systems and programming languages and ultimately the software engineers who have to develop the applications.
We first look at the concurrency primitives that modern operating systems provides us to work with, namely processes and threads. With Python, a common approach is to use threads which have shared memory. However this can be difficult to do well as it is easy to run into race conditions and other bugs. And Python threads, unlike OS threads, cannot run at the same time due to the infamous GIL, thus limiting us to only one core.
To utilize multiple cores in Python, an effective and better solution is to use multiple processes, each with their own memory space and provide inter-process communication using message passing. We will look at the multiprocessing module and how to use it for this purpose.
Μεταπρογραµµατισµός κώδικα Python σε γλώσσα γραµµικού χρόνου για αυτόµατη επα...ISSEL
Ο όρος επαλήθευση λογικής κατά την εκτέλεση οριοθετεί ένα πεδίο που εκτείνεται από τον έλεγχο λογισµικού για τη συµµόρφωση µε ένα σύνολο προδιαγραφών, έως την εναρµόνιση µε καλές λογικές πρακτικές κατά τη συγγραφή κώδικα. Στο πλαίσιο αυτό, υλοποιήσαµε τη lovpy, µια ϐιβλιοθήκη µεταπρογραµµατισµού για τη γλώσσα Python, που εισάγει σε αυτή τις δυνατότητες της επαλήθευσης λογικής κατά την εκτέλεση. Ο καθορισµός της πρότυπης λογικής γίνεται χρησιµοποιώντας τη διαισθητική γλώσσα έκφρασης προδιαγραφών Gherkin, ενώ η χρήση της ϐιβλιοθήκης δεν απαιτεί καµία αλλαγή στον υπάρχον κώδικα. Για την υλοποίησή της αξιοποιήσαµε µια σειρά εργαλείων της ϑεωρίας γράφων, της ϑεωρίας τυπικών γλωσσών, της χρονικής λογικής καθώς και µοντέλα ϐαθιάς µηχανικής µάθησης, εστιάζοντας περισσότερο στα νευρωνικά δίκτυα γράφων. Θεµελιώσαµε µαθηµατικά ένα νέο είδος γράφου για την αναπαράσταση χρονικών προδιαγραφών και ορίσαµε για αυτόν ένα σύνολο µαθηµατικά τεκµηριωµένων λογικών αλγορίθµων. Στη συνέχεια, αξιοποιήσαµε τις δοµές αυτές προκειµένου να υλοποιήσουµε ένα νέο σύστηµα αυτόµατης απόδειξης ϑεωρη µάτων, το οποίο µας εξασφαλίζει την απόλυτη εγκυρότητα των εντοπισµένων παραβιάσεων. Αξιολογήσαµε πέντε διαφορετικές αποδεικτικές αρχιτεκτονικές, αποτελούµενες από ευριστικούς κανόνες και απλά νευρωνικά µοντέλα, µέχρι µεγάλα νευρωνικά δίκτυα γράφων. Για την εκπαίδευση των νευρωνικών συστηµάτων υλοποιήσαµε ένα µηχανισµό παραγω γής συνθετικών ϑεωρηµάτων, αξιοποιώντας µια σειρά από µαθηµατικές ιδιότητες. Τέλος, χρησιµοποιήσαµε τη lovpy για να εντοπίσουµε σφάλµατα σε δύο δηµοφιλή ϐιβλιοθήκες ανοιχτού κώδικα, την Django και την Keras.
Understand and interpret concept the loop statements and give some examples. • Students can design programs involving loop statements. • Gain knowledge and understanding of function of the loops inside the program. • Students can compare between loop statements , Iteration. , Loop structure, While - Do while, For - Loop control statement (start, end and step).
, Control all loops by break & continue statements, Infinite loop,Explain the different loops available in language with examples, Differ between loop control statement and the infinite loop , by Eng.&Educator Osama Ghandour
(Given to the Vancouver Erlang and Ruby/Rails Meetup groups on May 19, 2009.)
Erlang is an up-and-coming language on the web scene. New libraries and frameworks are sprouting up at a rampant rate, and web giants Facebook and Twitter are using it to develop highly-scalable web applications.
This talk will introduce Erlang as a language and platform, summarize its strengths and weaknesses, and cover how you can use Erlang and Ruby together to conquer the web frontier.
Speaker Bio:
Ken Pratt has been developing software for the web for over 10 years. He fell in love with Ruby four years ago, but is still passionate about learning other languages and platforms. He has developed scalable web services for Electronic Arts, built Rails-based web applications since pre-1.0, and been featured in interactive art installations.
Understand and interpret concept the loop statements and give some examples. • Students can design programs involving loop statements. • Gain knowledge and understanding of function of the loops inside the program. • Students can compare between loop statements , Iteration. , Loop structure, While - Do while, For - Loop control statement (start, end and step).
, Control all loops by break & continue statements, Infinite loop,Explain the different loops available in language with examples, Differ between loop control statement and the infinite loop , by Eng.&Educator Osama Ghandour
(Given to the Vancouver Erlang and Ruby/Rails Meetup groups on May 19, 2009.)
Erlang is an up-and-coming language on the web scene. New libraries and frameworks are sprouting up at a rampant rate, and web giants Facebook and Twitter are using it to develop highly-scalable web applications.
This talk will introduce Erlang as a language and platform, summarize its strengths and weaknesses, and cover how you can use Erlang and Ruby together to conquer the web frontier.
Speaker Bio:
Ken Pratt has been developing software for the web for over 10 years. He fell in love with Ruby four years ago, but is still passionate about learning other languages and platforms. He has developed scalable web services for Electronic Arts, built Rails-based web applications since pre-1.0, and been featured in interactive art installations.
These are the slides used on ElixirBarcelona meetup about Distributed Elixir by Oscar de Arriba (@odarriba)
Video: https://www.youtube.com/watch?v=Y0bVDPrXguY
Examples: https://github.com/odarriba/distributed-elixir-example
Python for Delphi (P4D) is a set of free components that wrap up the Python DLL into Delphi and Lazarus (FPC). They let you easily execute Python scripts, create new Python modules and new Python types. You can create Python extensions as DLLs and much more like scripting. P4D provides different levels of functionality:
Low-level access to the python API
High-level bi-directional interaction with Python
Access to Python objects using Delphi custom variants (VarPyth.pas)
Wrapping of Delphi objects for use in python scripts using RTTI (WrapDelphi.pas)
Creating python extension modules with Delphi classes and functions
Generate Scripts in maXbox from Python Installation
Concurrent Programming with Ruby and Tuple Spacesluccastera
Ruby threads are limited due to the Global Interpreter Lock. Therefore, the best way to do parallel computing with Ruby is to use multiple processes but how do you get these processes to communicate?
This session will provide some strategies for handling multi-process communication in Ruby, with a focus on the use of TupleSpaces. A TupleSpace provides a repository of tuples that can be accessed concurrently to implement a Blackboard system. Ruby ships with a built-in implementation of a TupleSpace with the Rinda library.
During the session, Luc will demonstrate how to use Rinda and will highlight other libraries/projects that facilitate interprocess communication and parallel computing in Ruby.
Documentation: https://izumi.7mind.io/latest/release/doc/distage/
Github: https://github.com/pshirshov/izumi-r2
Pavel Shirshov - DIStage: purely functional programming without sacrificing modularity with modern dependency injection for Scala
- Modularity and its importance
- DI-like mechanisms and their issues in Scala
- Why people think that "DI doesn't compose with functional programming", and why that's not true
- Designing a staged DI, for wiring at runtime, at compile-time, or mixed
- Staging programs for reliability, power and performance
- The pains of supporting rich Scala types (incl. How to emulate kind-polymorphism in Scala 2)
- Garbage collection in DI for better tests and deployments
Pavel's bio: Language-agnostic software engineer, coding for 18 years,
10 years of hands-on commercial engineering experience.
Led a cluster orchestration team at Yandex, "the Russian Google"; implemented an internal orchestration solution, "ISS" (Scala/Java/C++), managing 50K+ physical hosts across 6 datacenters.
Today, Pavel owns Irish R&D company Septimal Mind.
6. Kata: Grep
1. Local file-system directory and its
subdirectories contains large
number of text files
2. Extract unobfuscated e-mail
addresses; together with people
names, if possible
3. Write the list of unique e-mails into
specified output file
8. What is the Coding Dojo?
Coding Dojo is a meeting where a
bunch of coders get together to work
on a programming challenge.
They are there to have fun and to
engage in Deliberate Practice in
order to improve their skills.
9. Roles
● Sensei
● Moderator
● Pair:
○ Pilot
○ Navigator
● Audience
10. Sensei
Responsible for:
● No long discussions without the code.
● Working code by the end of the iteration.
● Tests are written during the iteration.
● Green tests by the end of the iteration,
otherwise the next iteration begins by
making tests green.
● Progress on fulfilling the Kata.
12. Pair
Pair of two developers working and talking at
the PC in order to complete the Kata.
Pair consists of the pilot and the navigator.
The pilot codes and the navigator helps the
pilot to understand what has been done in
the previous iterations and what the pair
should do now.
13. Principles
Working on Kata
At the Dojo one can't discuss a form
without code, and one can't show
code without tests.
14. Principles
Timeboxing
Pair works during a fixed period of time.
Switching
The person who enters into the coding
timebox begins to act as a pilot. The person
who was the pilot during the previous
iteration switches to the position of the
navigator.
15. Respect
● Ringtones of mobile phones are off. A
person can answer a call, but he or she
must leave the room to do it.
● Only the pair at the PC can talk loud.
● Sensei can interfere and talk loud by his or
her own decision.
16. Respect
● All discussions away from the PC must be
made in the way not to spoil the pair’s
deep diving into the Kata.
● If a person wants to ask or suggest smth
to the pair, he or she should raise the
hand and wait for the moderator for
permit.
● Only one person from the observers can
talk at a time.
18. Working agreement
? Randori, Parisian, Pairs ?
● Applicability of unit and functional
tests
● Randori round time
● Switching: round-robin, by request
19. Kata: Grep - Erlang cluster
1. Join Erlang cluster
2. Send grep module to peers and
start grep worker there
3. Send messages with files content
4. Receive list of e-mail
5. Aggregate results
20. Join Erlang cluster
1. Ping LAN bootstrap nodes
- bootstrap@extensa
- bootstrap@hal9011
2. Workes are:
+ nodes()
- worker if name starts with "worker"
*except coordinator local VM [node()] to
simplify programming and prevent
aggregator bottlenecks
21. Send code to peers
1. The functionality is implemented by
module 'grep'
2. 'grep' is also an (Erlang) application
3. use code:get_object/1 to read the module
beam
4. use rpc:(multi)call/4, code:load_binary/1,
erlang:spawn/4 to load beam and start
application remotely
22. Send work
1. API signature:
grep:grep(Dir) ->
[ email1@domain, email2@domain2 ]
2. Obtain list of workers by calling nodes/0
3. Scan directory recursively and create a
tuple:
Work = { self(), Content }
4. Send message to next peer:
{ grep_worker, Node } ! { self(), Work }
23. Receive result
1. After all work units are sent - receive all
replies or timeout occur:
recv(N) -> c:flush(), recv([], N).
recv(A, 0) -> A.
recv(A, N) ->
receive
{emails, Mails} -> recv([Mails | A], N-1)
after 1000 -> A
end.
2. Flatten, sort, uniq results
24. Worker workflow with OTP
1. Use OTP gen_server and supervisor
behaviour-s with simple_one_for_one
strategy
2. application:start/1 calls grep:init/1
3. grep:init/1 spawns a number of workers
that is equal to core count as returned by
erlang:system_info(schedulers)
4. grep:init/1 registers all workers with global:
register_name/2
26. Latvian Developers Network
http://www.ldn.lv
http://fb.me/LatvianDevelopersNetwork
@LVDevNet
Arkadi Shislov
arkadi.shishlov@gmail.com
http://bitbucket.org/arkadi
Vladimir Tarasow
http://about.me/NetRat
27. This work is licensed under the Creative Commons Attribution-
NonCommercial-ShareAlike 3.0 Unported License. To view a copy of this
license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/