Slides originating from a talk I gave at ScalaMUC on 2013-12-17. The corresponding code can be found at https://github.com/afwlehmann/iteratee-tutorial .
A fast-paced introduction to TensorFlow 2 about some important new features (such as generators and the @tf.function decorator) and TF 1.x functionality that's been removed from TF 2 (yes, tf.Session() has retired).
Concise code samples are presented to illustrate how to use new features of TensorFlow 2. You'll also get a quick introduction to lazy operators (if you know FRP this will be super easy), along with a code comparison between TF 1.x/iterators with tf.data.Dataset and TF 2/generators with tf.data.Dataset.
Finally, we'll look at some tf.keras code samples that are based on TensorFlow 2. Although familiarity with TF 1.x is helpful, newcomers with an avid interest in learning about TensorFlow 2 can benefit from this session.
Gentlest Introduction to Tensorflow - Part 3Khor SoonHin
Articles:
* https://medium.com/all-of-us-are-belong-to-machines/gentlest-intro-to-tensorflow-part-3-matrices-multi-feature-linear-regression-30a81ebaaa6c
* https://medium.com/all-of-us-are-belong-to-machines/gentlest-intro-to-tensorflow-4-logistic-regression-2afd0cabc54
Video: https://youtu.be/F8g_6TXKlxw
Code: https://github.com/nethsix/gentle_tensorflow
In this part, we:
* Use Tensorflow for linear regression models with multiple features
* Use Tensorflow for logistic regression models with multiple features. Specifically:
* Predict multi-class/discrete outcome
* Explain why we use cross-entropy as cost function
* Explain why we use softmax
* Tensorflow Cheatsheet #1
* Single feature linear regression
* Multi-feature linear regression
* Multi-feature logistic regression
Introduction to Deep Learning, Keras, and TensorflowOswald Campesato
A fast-paced introduction to Deep Learning concepts, such as activation functions, cost functions, back propagation, and then a quick dive into CNNs. Basic knowledge of vectors, matrices, and derivatives is helpful in order to derive the maximum benefit from this session. Then we'll see how to create a Convolutional Neural Network in Keras, followed by a quick introduction to TensorFlow and TensorBoard.
A fast-paced introduction to TensorFlow 2 about some important new features (such as generators and the @tf.function decorator) and TF 1.x functionality that's been removed from TF 2 (yes, tf.Session() has retired).
Some concise code samples are presented to illustrate how to use new features of TensorFlow 2.
Scientific Computing with Python Webinar March 19: 3D Visualization with MayaviEnthought, Inc.
In this webinar, Didrik Pinte provides an introduction to MayaVi, the 3D interactive visualization library for the open source Enthought Tool Suite. These tools provide scientists and engineers a sophisticated Python development framework for analysis and visualization.
A fast-paced introduction to TensorFlow 2 regarding some important new features (such as generators and the @tf.function decorator), along with tf.data code samples and lazy operators. We'll also delve into the key ideas underlying CNNs, RNNs, and LSTMs, followed by some Keras-based code blocks.
This slide is used to do an introduction for the matplotlib library and this will be a very basic introduction. As matplotlib is a very used and famous library for machine learning this will be very helpful to teach a student with no coding background and they can start the plotting of maps from the ending of the slide by there own.
A fast-paced introduction to Deep Learning concepts, such as activation functions, cost functions, back propagation, and then a quick dive into CNNs, followed by a Keras code sample for defining a CNN. Basic knowledge of vectors, matrices, and derivatives is helpful in order to derive the maximum benefit from this session. Then we'll see a short introduction to TensorFlow 1.x and some insights into TF 2 that will be released some time this year.
This session for beginners introduces tf.data APIs for creating data pipelines by combining various "lazy operators" in tf.data, such as filter(), map(), batch(), zip(), flatmap(), take(), and so forth.
Familiarity with method chaining and TF2 is helpful (but not required). If you are comfortable with FRP, the code samples in this session will be very familiar to you.
Video: https://youtu.be/dYhrCUFN0eM
Article: https://medium.com/p/the-gentlest-introduction-to-tensorflow-248dc871a224
Code: https://github.com/nethsix/gentle_tensorflow/blob/master/code/linear_regression_one_feature.py
This alternative introduction to Google's official Tensorflow (TF) tutorial strips away the unnecessary concepts that overly complicates getting started. The goal is to use TF to perform Linear Regression (LR) that has only a single-feature. We show how to model the LR using a TF graph, how to define the cost function to measure how well the an LR model fits the dataset, and finally train the LR model to find the best fit model.
Web pages can get very complex and slow. In this talk, I share how we solve some of these problems at LinkedIn by leveraging composition and streaming in the Play Framework. This was my keynote for Ping Conference 2014 ( http://www.ping-conf.com/ ): the video is on ustream ( http://www.ustream.tv/recorded/42801129 ) and the sample code is on github ( https://github.com/brikis98/ping-play ).
A fast-paced introduction to TensorFlow 2 about some important new features (such as generators and the @tf.function decorator) and TF 1.x functionality that's been removed from TF 2 (yes, tf.Session() has retired).
Concise code samples are presented to illustrate how to use new features of TensorFlow 2. You'll also get a quick introduction to lazy operators (if you know FRP this will be super easy), along with a code comparison between TF 1.x/iterators with tf.data.Dataset and TF 2/generators with tf.data.Dataset.
Finally, we'll look at some tf.keras code samples that are based on TensorFlow 2. Although familiarity with TF 1.x is helpful, newcomers with an avid interest in learning about TensorFlow 2 can benefit from this session.
Gentlest Introduction to Tensorflow - Part 3Khor SoonHin
Articles:
* https://medium.com/all-of-us-are-belong-to-machines/gentlest-intro-to-tensorflow-part-3-matrices-multi-feature-linear-regression-30a81ebaaa6c
* https://medium.com/all-of-us-are-belong-to-machines/gentlest-intro-to-tensorflow-4-logistic-regression-2afd0cabc54
Video: https://youtu.be/F8g_6TXKlxw
Code: https://github.com/nethsix/gentle_tensorflow
In this part, we:
* Use Tensorflow for linear regression models with multiple features
* Use Tensorflow for logistic regression models with multiple features. Specifically:
* Predict multi-class/discrete outcome
* Explain why we use cross-entropy as cost function
* Explain why we use softmax
* Tensorflow Cheatsheet #1
* Single feature linear regression
* Multi-feature linear regression
* Multi-feature logistic regression
Introduction to Deep Learning, Keras, and TensorflowOswald Campesato
A fast-paced introduction to Deep Learning concepts, such as activation functions, cost functions, back propagation, and then a quick dive into CNNs. Basic knowledge of vectors, matrices, and derivatives is helpful in order to derive the maximum benefit from this session. Then we'll see how to create a Convolutional Neural Network in Keras, followed by a quick introduction to TensorFlow and TensorBoard.
A fast-paced introduction to TensorFlow 2 about some important new features (such as generators and the @tf.function decorator) and TF 1.x functionality that's been removed from TF 2 (yes, tf.Session() has retired).
Some concise code samples are presented to illustrate how to use new features of TensorFlow 2.
Scientific Computing with Python Webinar March 19: 3D Visualization with MayaviEnthought, Inc.
In this webinar, Didrik Pinte provides an introduction to MayaVi, the 3D interactive visualization library for the open source Enthought Tool Suite. These tools provide scientists and engineers a sophisticated Python development framework for analysis and visualization.
A fast-paced introduction to TensorFlow 2 regarding some important new features (such as generators and the @tf.function decorator), along with tf.data code samples and lazy operators. We'll also delve into the key ideas underlying CNNs, RNNs, and LSTMs, followed by some Keras-based code blocks.
This slide is used to do an introduction for the matplotlib library and this will be a very basic introduction. As matplotlib is a very used and famous library for machine learning this will be very helpful to teach a student with no coding background and they can start the plotting of maps from the ending of the slide by there own.
A fast-paced introduction to Deep Learning concepts, such as activation functions, cost functions, back propagation, and then a quick dive into CNNs, followed by a Keras code sample for defining a CNN. Basic knowledge of vectors, matrices, and derivatives is helpful in order to derive the maximum benefit from this session. Then we'll see a short introduction to TensorFlow 1.x and some insights into TF 2 that will be released some time this year.
This session for beginners introduces tf.data APIs for creating data pipelines by combining various "lazy operators" in tf.data, such as filter(), map(), batch(), zip(), flatmap(), take(), and so forth.
Familiarity with method chaining and TF2 is helpful (but not required). If you are comfortable with FRP, the code samples in this session will be very familiar to you.
Video: https://youtu.be/dYhrCUFN0eM
Article: https://medium.com/p/the-gentlest-introduction-to-tensorflow-248dc871a224
Code: https://github.com/nethsix/gentle_tensorflow/blob/master/code/linear_regression_one_feature.py
This alternative introduction to Google's official Tensorflow (TF) tutorial strips away the unnecessary concepts that overly complicates getting started. The goal is to use TF to perform Linear Regression (LR) that has only a single-feature. We show how to model the LR using a TF graph, how to define the cost function to measure how well the an LR model fits the dataset, and finally train the LR model to find the best fit model.
Web pages can get very complex and slow. In this talk, I share how we solve some of these problems at LinkedIn by leveraging composition and streaming in the Play Framework. This was my keynote for Ping Conference 2014 ( http://www.ping-conf.com/ ): the video is on ustream ( http://www.ustream.tv/recorded/42801129 ) and the sample code is on github ( https://github.com/brikis98/ping-play ).
Asynchronous web apps with the Play Framework 2.0Oscar Renalias
Brief introduction to the asynchronous and reactive IO capabilities available in Play 2.0.
Source code of the demos available here: https://github.com/oscarrenalias/wjax-2012-play-async-apps
The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaSphere ver)Eugene Yokota
Talk given at ScalaSphere 2017. A review of:
- the sbt 0.13.x series that’s been under development as a technology preview since 2014
- the sbt server feature which is planned to be shipped with the next major sbt release
- and the rest of sbt 1.0
Detecting Hacks: Anomaly Detection on Networking DataJames Sirota
See https://medium.com/@jamessirota for a series of blog entries that goes with this deck...
Defense in Depth for Big Data
Network Anomaly Detection Overview
Volume Anomaly Detection
Feature Anomaly Detection
Model Architecture
Deployment on OpenSOC Platform
Questions
Video: http://joyent.com/blog/linux-performance-analysis-and-tools-brendan-gregg-s-talk-at-scale-11x ; This talk for SCaLE11x covers system performance analysis methodologies and the Linux tools to support them, so that you can get the most out of your systems and solve performance issues quickly. This includes a wide variety of tools, including basics like top(1), advanced tools like perf, and new tools like the DTrace for Linux prototypes.
Function Programming in Scala.
A lot of my examples here comes from the book
Functional programming in Scala By Paul Chiusano and Rúnar Bjarnason, It is a good book, buy it.
19. Data Structures and Algorithm ComplexityIntro C# Book
In this chapter we will compare the data structures we have learned so far by the performance (execution speed) of the basic operations (addition, search, deletion, etc.). We will give specific tips in what situations what data structures to use. We will explain how to choose between data structures like hash-tables, arrays, dynamic arrays and sets implemented by hash-tables or balanced trees. Almost all of these structures are implemented as part of NET Framework, so to be able to write efficient and reliable code we have to learn to apply the most appropriate structures for each situation.
Functional programming in Scala. Looking at various examples of defining a program first and executing it at some later stage, separating pure functions from side effects.
In this presentation, You will get to know about Function Literal,Higher Order Function,Partial Function,Partial Applied Function,Nested Function,Closures.
This fast-paced session starts with an introduction to neural networks and linear regression models, along with a quick view of TensorFlow, followed by some Scala APIs for TensorFlow. You'll also see a simple dockerized image of Scala and TensorFlow code and how to execute the code in that image from the command line. No prior knowledge of NNs, Keras, or TensorFlow is required (but you must be comfortable with Scala).
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
2. History
Iterator pattern
Iteratees
Discussion
Brief history
2008 Oleg Kiselyov, “Incremental multi-level input
processing with left-fold enumerator”, DEFUN 2008
2010-05-12 John W. Lato, “Teaching an Old Fool New Tricks’,
Monad Reader #16
´
2010-05-26 Available in scalaz 5.0 R´nar Oli
u
2012-03-13 Initial release of the play framework 2.0
2/29
3. History
Iterator pattern
Iteratees
Discussion
We’ve often seen something along the lines of
import io.Source.fromFile
val it: Iterator[String] = fromFile("foo.txt").getLines
var result = 0
while (it.hasNext) {
val line = it.next
result += line.toInt
}
// Do something with the result
3/29
4. History
Iterator pattern
Iteratees
Discussion
Issues
Repetitive pattern (DRY principle)
Manual pulling
Mutability, imperative style
No error handling (we sometimes just forget, right?)
No (one|two)-way communication (Exceptions don’t count)
Resource handling (how long do we need a resource and who
is responsible for opening/closing/recovering it?)
Missing or rather difficult composability
What if the input stream were infinite?
4/29
5. History
Iterator pattern
Iteratees
Discussion
Try to be more “functional” and invert control...
val it: Iterator[String] = fromFile(foo.txt).getLines
var result = 0
it foreach { line =
result += line.toInt
}
5/29
6. History
Iterator pattern
Iteratees
Discussion
define ourselves a re-usable utility function...
def enum(it: Iterator[String]): Int = {
var result = 0
it foreach { line = result += line.toInt }
result
}
val foo = enum(fromFile(foo.txt).getLines)
6/29
7. History
Iterator pattern
Iteratees
Discussion
being more versatile for the greater good...
def enum(it: Iterator[String])(init: Int)
(f: (Int, String) = Int): Int = {
var result = init
it foreach { line = result = f(result, line) }
result
}
val it: Iterator[String] = fromFile(foo.txt).getLines
val foo: Int = enum(it)(0)(_ + _.toInt)
7/29
8. History
Iterator pattern
Iteratees
Discussion
possibly even generic...
def enum[In,Out](it: Iterator[In])(init: Out)
(f: (Out, In) = Out): Out = {
var result = init
it foreach { x = result = f(result, x) }
result
}
val it: Iterator[String] = fromFile(foo.txt).getLines
val foo: Int = enum(it)(0)(_ + _.toInt)
8/29
9. History
Iterator pattern
Iteratees
Discussion
provide sophisticated error handling...
def enum[In,Out](it: Iterator[In])(init: Out)
(f: (Out, In) = Out): Out = {
var result = init
try {
it foreach { x = result = f(result, x) }
} catch {
case t: Throwable = /* TODO (fingers crossed) */
}
result
}
val it: Iterator[String] = fromFile(foo.txt).getLines
val foo: Int = enum(it)(0)(_ + _.toInt)
9/29
10. History
Iterator pattern
Iteratees
Discussion
or rather use {your-favorite-“monad”-here} for error handling,
asynchronism and composability...
def enum[In,Out](it: Iterator[In])(init: Out)
(f: (Out, In) = Out): Option[Out] = {
try {
var result = init
it foreach { x = result = f(result, x) }
Some(result)
} catch {
case t: Throwable = None
}
}
val it: Iterator[String] = fromFile(foo.txt).getLines
val foo: Option[Int] = enum(it)(0)(_ + _.toInt)
10/29
11. History
Iterator pattern
Iteratees
Discussion
only to realize we’ve already had it all!?
val it: Iterator[String] = fromFile(foo.txt).getLines
val foo: Try[Int] = it.foldLeft(Try(0)) {
// f: (Out, In) = Out
case (acc, line) =
acc map { x = x + line.toInt }
}
11/29
12. History
Iterator pattern
Iteratees
Discussion
only to realize we’ve already had it all!?
val it: Iterator[String] = fromFile(foo.txt).getLines
val foo: Try[Int] = it.foldLeft(Try(0)) {
// f: (Out, In) = Out
case (acc, line) =
acc map { x = x + line.toInt }
}
Really? What about
producer and consumer talking back and forth
cannot cope with infinite streams
resource handling (stick this into another function?)
asynchronism (could have used Futures)
data which are not available all at once
11/29
13. History
Iterator pattern
Iteratees
Discussion
only to realize we’ve already had it all!?
val it: Iterator[String] = fromFile(foo.txt).getLines
val foo: Try[Int] = it.foldLeft(Try(0)) {
// f: (Out, In) = Out
case (acc, line) =
acc map { x = x + line.toInt }
}
Think ofWhat about
Really?
producer and consumer talking back and forth
cannot cope with infinite streams
foldLeft as Enumerator
resource handling (stickIteratee another function?)
acc together with f as this into
asynchronism (could have used Futures)
data which are not available all at once
11/29
14. History
Iterator pattern
Iteratees
Discussion
Enumerators
Basics
Composition
Enumeratees
Enumerators . . .
are stream producers
push subsequent chunks of data to an Iteratee, hence folding
the Iteratee over the input stream
sealed trait Enumerator[F] {
def apply[T](iter: Iteratee[F,T]): Iteratee[F,T]
def run[T](iter: Iteratee[F,T]): T
}
act synchronously or asynchronously (think Futures)
come with batteries included
object Enumerator {
def apply[T](xs: T*): Enumerator[T]
def fromTextFile(fileName: String): Enumerator[String]
def fromStream(s: InputStream, chunkSize: Int = 8192)
: Enumerator[Array[Byte]]
}
12/29
17. History
Iterator pattern
Iteratees
Discussion
Enumerators
Basics
Composition
Enumeratees
Iteratees . . .
stream consumers
consume chunks of input (as a whole)
are immutable, re-usable computations
state is encoded through type
sealed trait Iteratee[F,T] { def run: T }
case class Done[F,T] (result: T, remainingInput: Input[F])
case class Cont[F,T] (k: Input[F] = Iteratee[F,T])
case class Error[F,T](t: Throwable)
// All of Done, Cont and Error extend Iteratee[F,T]
14/29
19. History
Iterator pattern
Iteratees
Discussion
Enumerators
Basics
Composition
Enumeratees
So far:
Enumerators fold iteratees over a stream
Enumerators communicate state through Input[T]
Iteratees encapsulate result, error or computation
Let’s revisit our “enumerator”:
def enum[F,T](xs: List[F])(it: Iteratee[F,T]): Iteratee[F,T] =
(xs, it) match {
case (h::t, Cont(k)) = enum(t)( k(Element(h)) )
case _
= it
}
// k: Input[In] = Iteratee[F,T]
This is all we need to have some working examples.
15/29
25. History
Iterator pattern
Iteratees
Discussion
Enumerators
Basics
Composition
Enumeratees
Wouldn’t it be awesome if Iteratees were composable?
def drop1Keep1[T] = for {
_ - drop[T](1)
h - head
} yield h
def pair[T] = for {
h1 - head[T]
h2 - head
} yield (h1, h2)
They are indeed! Iteratees compose sequentially (and so do
Enumerators).
scala enum(Hello World!.toList)(drop1Keep1).run
res1: Char = e
scala enum(Hello World!.toList)(pair).run
res2: (Char, Char) = (H,e)
20/29
32. History
Iterator pattern
Iteratees
Discussion
Brief discussion:
Enumerators, Iteratees, Enumeratees
Easy to use
Composable in sequence and in parallel
Tail recursion ⇒ no stack overflow
Thin layer ⇒ high performance
Good match for asynchronuous environments, e.g. the play
framework
Pure form lacks functional treatment of side effects
26/29
33. History
Iterator pattern
Iteratees
Discussion
Where to go from here
Read blogs tutorials, start out with [1] to [6]
Use or rather implement iteratees
Check out the play framework
Asynchronous iteratees
Numerous factory methods, e.g. easily bridge
between Rx’s Observables and Enumerators [3]
All the other goodies from play
Check out scalaz 7
Implementation in terms of monad transformers
⇒ your choice of IO, Future, Option, . . .
All what’s missing in the standard library
See examples and explanations at [1] and [2]
27/29
34. History
Iterator pattern
Iteratees
Discussion
Thank you for your attention
Anyone trying to understand monads will inevitably run
into the [...] monad, and the results are almost always
the same: bewilderment, confusion, anger, and ultimately
Perl.
Daniel Spiewak, Dec. 2010
28/29
35. History
Iterator pattern
Iteratees
Discussion
References:
[1] “Enumeration-based I/O With Iteratees” @
http://blog.higher-order.com/blog/2010/10/14/scalaz-tutorial-enumeration-based-io-with-iteratees/
[2] “learning Scalaz: Enumeration-Based I/O with Iteratees” @
http://eed3si9n.com/learning-scalaz/Iteratees.html
[3] “RxPlay: Diving into iteratees and observables and making
them place nice” @
http://bryangilbert.com/code/2013/10/22/rxPlay-making-iteratees-and-observables-play-nice/
[4] “Understanding Play2 Iteratees for Normal Humans” @
http://mandubian.com/2012/08/27/understanding-play2-iteratees-for-normal-humans/
[5] “Incremental multi-level input processing and collection
enumeration” @ http://okmij.org/ftp/Streams.html
[6] “Teaching an Old Fool New Tricks” @
http://themonadreader.wordpress.com/2010/05/12/issue-16/
29/29
37. Running Iteratees
Example
run makes great effort to yield the final result:
def run: T = this match {
case Done(rslt, _) = rslt
case Error(t)
= throw t
// k: Input[F] = Iteratee[F,T]
case Cont(k)
= k(EOF).run
// may loop forever
}
31/29
38. Running Iteratees
Example
run makes great effort to yield the final result:
def run: T = this match {
case Done(rslt, _)
= rslt
case Error(t)
= throw t
// k: Input[F] = Iteratee[F,T]
case Cont(k)
= k(EOF) match {
case Done(rslt, _) = rslt
case Cont(_)
= sys.error(Diverging iteratee!)
case Error(t)
= throw t
}
}
31/29