This document discusses a presentation on advanced uses of generators in Python. It covers context managers, which allow entry and exit actions for code blocks using the 'with' statement. Generators can be used to implement context managers via the yield keyword and decorator contextmanager. This transforms generator functions into objects that support the required __enter__ and __exit__ methods to monitor code block execution. The presentation aims to expand understanding of generators beyond iteration by exploring this technique for simplifying resource management tasks like file handling through context managers.
Walter Heck, founder of OlinData, presented a step-by-step guide on how to set up a proper puppet repository, complete with the brand new PuppetDB, exported resources and usage of open source modules.
Facebook is a company that operates at massive scale. In this talk we’ll talk about how we use Python at Facebook.
Be it building back-end services, fast prototyping, automation, scaling operations, or simply gluing together various pieces of our infrastructure, Python is at the heart of it and allows our engineers to quickly deliver working solutions.
We’ll talk about our review process, unit testing, deployment workflow and various open-source framework we use.
A talk given at the August 2010 meeting of the Linux Users of Victoria. About using their mailing list of some 20,000 messages (since the start of 2007) with over 2 million words, as a demonstration of using a web corpus in NLTK (Natural Language Tool Kit), the Python library.
Shared Object images in Docker: What you need is what you want.Workhorse Computing
Docker images require appropriate shared object files (".so") to run. Rather than assume Ubuntu has the correct lib's, use ldd to get a list and install the ones you know you need. This can reduce the underlying images from GB to a few MB.
The PyConTW (http://tw.pycon.org) organizer wishes to improve the quality and quantity of the programming cummunities in Taiwan. Though Python is their core tool and methodology, they know it's worth to learn and communicate with wide-ranging communities. Understanding cultures and ecosystem of a language takes me about three to six months. This six-hour course wraps up what I - an experienced Java developer - have learned from Python ecosystem and the agenda of the past PyConTW.
你可以在以下鏈結找到中文內容:
http://www.codedata.com.tw/python/python-tutorial-the-1st-class-1-preface
Containers for Science and High-Performance ComputingDmitry Spodarets
Within this talk, we will explore how Singularity liberates non-privileged users and host resources (such as interconnects, resource managers, file systems, accelerators, etc.) allowing users to take full control to set-up and run in their native environments. This talk explores how Singularity combines software packaging models with minimalistic containers to create very lightweight application bundles which can be simply executed and contained completely within their environment or be used to interact directly with the host file systems at native speeds. A Singularity application bundle can be as simple as containing a single binary application or as complicated as containing an entire workflow and is as flexible as you will need.
Sphinx autodoc - automated api documentation - PyCon.KR 2015Takayuki Shimizukawa
Using the automated documentation feature of Sphinx, you can make with ease the extensive documentation of Python program.
You just write python function documents (docstrings), Sphinx organizes them into the document, can be converted to a variety of formats.
In this session, I'll explain a documentation procedure that uses with sphinx autodoc and autosummary extensions.
Slides from Advaned Python lectures I gave recently in Haifa Linux club
Advanced python, Part 2:
- Slots vs Dictionaries
- Basic and Advanced Generators
- Async programming
Walter Heck, founder of OlinData, presented a step-by-step guide on how to set up a proper puppet repository, complete with the brand new PuppetDB, exported resources and usage of open source modules.
Facebook is a company that operates at massive scale. In this talk we’ll talk about how we use Python at Facebook.
Be it building back-end services, fast prototyping, automation, scaling operations, or simply gluing together various pieces of our infrastructure, Python is at the heart of it and allows our engineers to quickly deliver working solutions.
We’ll talk about our review process, unit testing, deployment workflow and various open-source framework we use.
A talk given at the August 2010 meeting of the Linux Users of Victoria. About using their mailing list of some 20,000 messages (since the start of 2007) with over 2 million words, as a demonstration of using a web corpus in NLTK (Natural Language Tool Kit), the Python library.
Shared Object images in Docker: What you need is what you want.Workhorse Computing
Docker images require appropriate shared object files (".so") to run. Rather than assume Ubuntu has the correct lib's, use ldd to get a list and install the ones you know you need. This can reduce the underlying images from GB to a few MB.
The PyConTW (http://tw.pycon.org) organizer wishes to improve the quality and quantity of the programming cummunities in Taiwan. Though Python is their core tool and methodology, they know it's worth to learn and communicate with wide-ranging communities. Understanding cultures and ecosystem of a language takes me about three to six months. This six-hour course wraps up what I - an experienced Java developer - have learned from Python ecosystem and the agenda of the past PyConTW.
你可以在以下鏈結找到中文內容:
http://www.codedata.com.tw/python/python-tutorial-the-1st-class-1-preface
Containers for Science and High-Performance ComputingDmitry Spodarets
Within this talk, we will explore how Singularity liberates non-privileged users and host resources (such as interconnects, resource managers, file systems, accelerators, etc.) allowing users to take full control to set-up and run in their native environments. This talk explores how Singularity combines software packaging models with minimalistic containers to create very lightweight application bundles which can be simply executed and contained completely within their environment or be used to interact directly with the host file systems at native speeds. A Singularity application bundle can be as simple as containing a single binary application or as complicated as containing an entire workflow and is as flexible as you will need.
Sphinx autodoc - automated api documentation - PyCon.KR 2015Takayuki Shimizukawa
Using the automated documentation feature of Sphinx, you can make with ease the extensive documentation of Python program.
You just write python function documents (docstrings), Sphinx organizes them into the document, can be converted to a variety of formats.
In this session, I'll explain a documentation procedure that uses with sphinx autodoc and autosummary extensions.
Slides from Advaned Python lectures I gave recently in Haifa Linux club
Advanced python, Part 2:
- Slots vs Dictionaries
- Basic and Advanced Generators
- Async programming
Python's "batteries included" philosophy means that it comes with an astonishing amount of great stuff. On top of that, there's a vibrant world of third-party libraries that help make Python even more wonderful. We'll go on a breezy, example-filled tour through some of my favorites, from treasures in the standard library to great third-party packages that I don't think I could live without, and we'll touch on some of the fuzzier aspects of the Python culture that make it such a joy to be part of.
Highlighted notes of:
Introduction to CUDA C: NVIDIA
Author: Blaise Barney
From: GPU Clusters, Lawrence Livermore National Laboratory
https://computing.llnl.gov/tutorials/linux_clusters/gpu/NVIDIA.Introduction_to_CUDA_C.1.pdf
Blaise Barney is a research scientist at Lawrence Livermore National Laboratory.
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...Athens Big Data
Title: MLOps Workshop: The Full ML Lifecycle - How to Use ML in Production
Speakers: Spyros Cavadias (https://www.linkedin.com/in/spyros-cavadias/), Konstantinos Pittas (https://www.linkedin.com/in/konstantinos-pittas-83310270/), Thanos Gkinakos (https://www.linkedin.com/in/thanos-gkinakos-03582a128/)
Date: Saturday, December 17, 2022
Event: https://www.meetup.com/athens-big-data/events/289927468/
Docker has created enormous buzz in the last few years. Docker is a open-source software containerization platform. It provides an ability to package software into standardised units on Docker for software development. In this hands-on introductory session, I introduce the concept of containers, provide an overview of Docker, and take the participants through the steps for installing Docker. The main session involves using Docker CLI (Command Line Interface) - all the concepts such as images, managing containers, and getting useful work done is illustrated step-by-step by running commands.
TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...Fons Sonnemans
Tijdens deze sessie zal met behulp van demo's en voorbeeld code getoond worden van wat de nieuwe mogelijkheden zijn van ASP.NET MVC6 en Entity Framework Core 1.0. Hoe genereer je de model classes, de controllers en de views. Wat kan en dien je daarna nog aan te passen. Wat zijn de mogelijkheden voor validatie en weergaves. Hoe kan je de nieuwe TagHelpers slim toepassen en zelf ook maken. Hoe gebruik je async controllers en views om de schaalbaarheid en soms ook de performance te verbeteren.
Workshop at TYPO3 Developer Days 2018. Goal is to have a working ddev setup with a simple TYPO3 installation. A quick look in the extensive possibilities is also part of the presentation.
JavaScript is evolving. It’s an exciting time to be involved with this ubiquitous language of the web. Every year, we get exciting new features landing as part of the language. Let’s explore the freshly released features that were part of the 2019 ECMAScript specification. We’ll also briefly explore the process of how new features get proposed and added to the language, as well as the leading contenders expected to land in ES2020.
Creating sub zero dashboard plugin for apex with googleRoel Hartman
Presentation about how to create a Google Visualization PlugIn for Oracle Application Express. Presentation given at ODTUG 2010, APEXPosed 2010 and UKOUG 2010.
A practical step-by-step guide to Git, taking you through each phase of a project and explaining the use of Git at each step of the development process. Expect lots of how-to, but also some how-not-to, to avoid going down the wrong path.
The OSI Superboard II was the computer on which I first learned to program back in 1979. Python is why programming remains fun today. In this tale of old meets new, I describe how I have used Python 3 to create a cloud computing service for my still-working Superboard--a problem complicated by it only having 8Kb of RAM and 300-baud cassette tape audio ports for I/O.
In the ever-evolving landscape of technology, enterprise software development is undergoing a significant transformation. Traditional coding methods are being challenged by innovative no-code solutions, which promise to streamline and democratize the software development process.
This shift is particularly impactful for enterprises, which require robust, scalable, and efficient software to manage their operations. In this article, we will explore the various facets of enterprise software development with no-code solutions, examining their benefits, challenges, and the future potential they hold.
Understanding Nidhi Software Pricing: A Quick Guide 🌟
Choosing the right software is vital for Nidhi companies to streamline operations. Our latest presentation covers Nidhi software pricing, key factors, costs, and negotiation tips.
📊 What You’ll Learn:
Key factors influencing Nidhi software price
Understanding the true cost beyond the initial price
Tips for negotiating the best deal
Affordable and customizable pricing options with Vector Nidhi Software
🔗 Learn more at: www.vectornidhisoftware.com/software-for-nidhi-company/
#NidhiSoftwarePrice #NidhiSoftware #VectorNidhi
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppGoogle
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-fusion-buddy-review
AI Fusion Buddy Review: Key Features
✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini
✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique!
✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs!
✅Fully automated AI articles bulk generation!
✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more.
✅With one keyword or URL, generate complete websites, landing pages, and more…
✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7.
✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches.
✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all!
✅Save over $5000 per year and kick out dependency on third parties completely!
✅Brand New App: Not available anywhere else!
✅ Beginner-friendly!
✅ZERO upfront cost or any extra expenses
✅Risk-Free: 30-Day Money-Back Guarantee!
✅Commercial License included!
See My Other Reviews Article:
(1) AI Genie Review: https://sumonreview.com/ai-genie-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
#AIFusionBuddyReview,
#AIFusionBuddyFeatures,
#AIFusionBuddyPricing,
#AIFusionBuddyProsandCons,
#AIFusionBuddyTutorial,
#AIFusionBuddyUserExperience
#AIFusionBuddyforBeginners,
#AIFusionBuddyBenefits,
#AIFusionBuddyComparison,
#AIFusionBuddyInstallation,
#AIFusionBuddyRefundPolicy,
#AIFusionBuddyDemo,
#AIFusionBuddyMaintenanceFees,
#AIFusionBuddyNewbieFriendly,
#WhatIsAIFusionBuddy?,
#HowDoesAIFusionBuddyWorks
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
1. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators:
The Final Frontier
David Beazley (@dabeaz)
http://www.dabeaz.com
Presented at PyCon'2014, Montreal
1
2. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
2
• GeneratorTricks for Systems Programmers (2008)
http://www.dabeaz.com/generators/
• A flying leap into generator awesomeness
3. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
3
• A Curious Course on Coroutines and Concurrency (2009)
http://www.dabeaz.com/coroutines/
• Wait, wait? There's more than iteration?
4. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Today's Installment
4
• Everything else you ever wanted to know
about generators, but were afraid to try
• Part 3 of a trilogy
5. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Requirements
5
• You need Python 3.4 or newer
• No third party extensions
• Code samples and notes
http://www.dabeaz.com/finalgenerator/
• Follow along if you dare!
6. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Disclaimer
6
• This is an advanced tutorial
• Assumes general awareness of
• Core Python language features
• Iterators/generators
• Decorators
• Common programming patterns
• I learned a LOT preparing this
7. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Will I Be Lost?
7
• Although this is the third part of a series, it's
mostly a stand-alone tutorial
• If you've seen prior tutorials, that's great
• If not, don't sweat it
• Be aware that we're focused on a specific use of
generators (you just won't get complete picture)
8. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Focus
8
• Material in this tutorial is probably not
immediately applicable to your day job
• More thought provoking and mind expanding
• from __future__ import future
practical
utility
bleeding
edge
9. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part I
9
Preliminaries - Generators and Coroutines
(rock)
10. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators 101
• yield statement defines a generator function
10
def countdown(n):
while n > 0:
yield n
n -= 1
• You typically use it to feed iteration
for x in countdown(10):
print('T-minus', x)
• A simple, yet elegant idea
11. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Under the Covers
11
• Generator object runs in response to next()
>>> c = countdown(3)
>>> c
<generator object countdown at 0x10064f900>
>>> next(c)
3
>>> next(c)
2
>>> next(c)
1
>>> next(c)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
>>>
• StopIteration raised when function returns
12. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
12
• Generators as "iterators" misses the big picture
• There is so much more to yield
13. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators as Pipelines
• Stacked generators result in processing pipelines
• Similar to shell pipes in Unix
13
generator
input
sequence
for x in s:generator generator
def process(sequence):
for s in sequence:
... do something ...
yield item
• Incredibly useful (see prior tutorial)
14. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Coroutines 101
• yield can receive a value instead
14
def receiver():
while True:
item = yield
print('Got', item)
• It defines a generator that you send things to
recv = receiver()
next(recv) # Advance to first yield
recv.send('Hello')
recv.send('World')
16. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Fundamentals
• The yield statement defines a generator function
16
def generator():
...
... yield ...
...
• The mere presence of yield anywhere is enough
• Calling the function creates a generator instance
>>> g = generator()
>>> g
<generator object generator at 0x10064f120>
>>>
17. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advancing a Generator
• next(gen) - Advances to the next yield
17
def generator():
...
...
yield item
...
• Returns the yielded item (if any)
• It's the only allowed operation on a newly
created generator
• Note: Same as gen.__next__()
18. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Sending to a Generator
• gen.send(item) - Send an item to a generator
18
def generator():
...
item = yield
...
...
yield value
• Wakes at last yield, returns sent value
• Runs to the next yield and emits the value
g = generator()
next(g) # Advance to yield
value = g.send(item)
19. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Closing a Generator
• gen.close() - Terminate a generator
19
def generator():
...
try:
yield
except GeneratorExit:
# Shutting down
...
• Raises GeneratorExit at the yield
• Only allowed action is to return
• If uncaught, generator silently terminates
g = generator()
next(g) # Advance to yield
g.close() # Terminate
20. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Raising Exceptions
• gen.throw(typ [, val [,tb]]) - Throw exception
20
def generator():
...
try:
yield
except RuntimeError as e:
...
...
yield val
• Raises exception at yield
• Returns the next yielded value (if any)
g = generator()
next(g) # Advance to yield
val = g.throw(RuntimeError,
'Broken')
21. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator ReturnValues
• StopIteration raised on generator exit
21
def generator():
...
yield
...
return result
• Return value (if any) passed with exception
• Note: Python 3 only behavior (in Python 2,
generators can't return values)
g = generator()
try:
next(g)
except StopIteration as e:
result = e.value
22. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Delegation
• yield from gen - Delegate to a subgenerator
22
def generator():
...
yield value
...
return result
def func():
result = yield from generator()
• Allows generators to call other generators
• Operations take place at the current yield
• Return value (if any) is returned
23. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Delegation Example
• Chain iterables together
23
def chain(x, y):
yield from x
yield from y
• Example:
>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> for x in chain(a, b):
... print(x,end=' ')
...
1 2 3 4 5 6
>>> c = [7,8,9]
>>> for x in chain(a, chain(b, c)):
... print(x, end=' ')
...
1 2 3 4 5 6 7 8 9
>>>
24. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Mini-Reference
• Generator instance operations
24
gen = generator()
next(gen) # Advance to next yield
gen.send(item) # Send an item
gen.close() # Terminate
gen.throw(exc, val, tb) # Raise exception
result = yield from gen # Delegate
• Using these, you can do a lot of neat stuff
• Generator definition
def generator():
...
yield
...
return result
25. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 2
25
and now for something completely different
26. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Common Motif
• Consider the following
26
f = open()
...
f.close()
lock.acquire()
...
lock.release()
db.start_transaction()
...
db.commit()
start = time.time()
...
end = time.time()
• It's so common, you'll see it everywhere!
27. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Managers
• The 'with' statement
27
with open(filename) as f:
statement
statement
...
with lock:
statement
statement
...
• Allows control over entry/exit of a code block
• Typical use: everything on the previous slide
28. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• It's easy to make your own (@contextmanager)
28
import time
from contextlib import contextmanager
@contextmanager
def timethis(label):
start = time.time()
try:
yield
finally:
end = time.time()
print('%s: %0.3f' % (label, end-start)
• This times a block of statements
29. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
29
• Usage
with timethis('counting'):
n = 1000000
while n > 0:
n -= 1
• Output
counting: 0.023
30. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• Another example: temporary directories
30
import tempfile, shutil
from contextlib import contextmanager
@contextmanager
def tempdir():
outdir = tempfile.mkdtemp()
try:
yield outdir
finally:
shutil.rmtree(outdir)
• Example
with tempdir() as dirname:
...
31. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Whoa,Whoa, Stop!
• Another example: temporary directories
31
import tempfile, shutil
from contextlib import contextmanager
@contextmanager
def tempdir():
outdir = tempfile.mkdtemp()
try:
yield outdir
finally:
shutil.rmtree(outdir)
• Example
with tempdir() as dirname:
...
What is this?
• Not iteration
• Not dataflow
• Not concurrency
• ????
32. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• Under the covers
32
with obj:
statements
statements
statements
...
statements
• If an object implements these methods it can
monitor entry/exit to the code block
obj.__enter__()
obj.__exit__()
33. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager
• Implementation template
33
class Manager(object):
def __enter__(self):
return value
def __exit__(self, exc_type, val, tb):
if exc_type is None:
return
else:
# Handle an exception (if you want)
return True if handled else False
• Use:
with Manager() as value:
statements
statements
34. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager Example
• Automatically deleted temp directories
34
import tempfile
import shutil
class tempdir(object):
def __enter__(self):
self.dirname = tempfile.mkdtemp()
return self.dirname
def __exit__(self, exc, val, tb):
shutil.rmtree(self.dirname)
• Use:
with tempdir() as dirname:
...
35. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Alternate Formulation
• @contextmanager is just a reformulation
35
import tempfile, shutil
from contextlib import contextmanager
@contextmanager
def tempdir():
dirname = tempfile.mkdtemp()
try:
yield dirname
finally:
shutil.rmtree(dirname)
• It's the same code, glued together differently
36. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• How does it work?
36
@contextmanager
def tempdir():
dirname = tempfile.mkdtemp()
try:
yield dirname
finally:
shutil.rmtree(dirname)
• Think of "yield" as scissors
• Cuts the function in half
37. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• Each half maps to context manager methods
37
@contextmanager
def tempdir():
dirname = tempfile.mkdtemp()
try:
yield dirname
statements
statements
statements
...
finally:
shutil.rmtree(dirname)
__enter__
__exit__
user statements ('with' block)
• yield is the magic that makes it possible
38. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• There is a wrapper class (Context Manager)
38
class GeneratorCM(object):
def __init__(self, gen):
self.gen = gen
def __enter__(self):
...
def __exit__(self, exc, val, tb):
...
• And a decorator
def contextmanager(func):
def run(*args, **kwargs):
return GeneratorCM(func(*args, **kwargs))
return run
39. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• enter - Run the generator to the yield
39
class GeneratorCM(object):
def __init__(self, gen):
self.gen = gen
def __enter__(self):
return next(self.gen)
def __exit__(self, exc, val, tb):
...
• It runs a single "iteration" step
• Returns the yielded value (if any)
40. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• exit - Resumes the generator
40
class GeneratorCM(object):
...
def __exit__(self, etype, val, tb):
try:
if etype is None:
next(self.gen)
else:
self.gen.throw(etype, val, tb)
raise RuntimeError("Generator didn't stop")
except StopIteration:
return True
except:
if sys.exc_info()[1] is not val: raise
• Either resumes it normally or raises exception
41. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Full Disclosure
• Actual implementation is more complicated
• There are some nasty corner cases
• Exceptions with no associated value
• StopIteration raised inside a with-block
• Exceptions raised in context manager
• Read source and see PEP-343
41
42. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Discussion
• Why start with this example?
• A completely different use of yield
• Being used to reformulate control-flow
• It simplifies programming for others (easy
definition of context managers)
• Maybe there's more... (of course there is)
42
43. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
43
Call me, maybe
44. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
44
Call me, maybe
45. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Async Processing
• Consider the following execution model
45
def func(args):
...
...
...
return result
run_async(func, args)
main thread
• Examples: Run in separate process or thread,
time delay, in response to event, etc.
46. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example:Thread Pool
46
from concurrent.futures import ThreadPoolExecutor
def func(x, y):
'Some function. Nothing too interesting'
import time
time.sleep(5)
return x + y
pool = ThreadPoolExecutor(max_workers=8)
fut = pool.submit(func, 2, 3)
r = fut.result()
print('Got:', r)
• Runs the function in a separate thread
• Waits for a result
47. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
47
>>> fut = pool.submit(func, 2, 3)
>>> fut
<Future at 0x1011e6cf8 state=running>
>>>
• Future - A result to be computed later
• You can wait for the result to return
>>> fut.result()
5
>>>
• However, this blocks the caller
48. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
48
def run():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(result_handler)
def result_handler(fut):
result = fut.result()
print('Got:', result)
• Alternatively, you can register a callback
• Triggered upon completion
49. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exceptions
49
>>> fut = pool.submit(func, 2, 'Hello')
>>> fut.result()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/concurrent/futures/_base.py",
line 395, in result
return self.__get_result()
File "/usr/local/lib/python3.4/concurrent/futures/_base.py",
line 354, in __get_result
raise self._exception
File "/usr/local/lib/python3.4/concurrent/futures/thread.py",
line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "future2.py", line 6, in func
return x + y
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>>
50. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures w/Errors
50
def run():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(result_handler)
def result_handler(fut):
try:
result = fut.result()
print('Got:', result)
except Exception as e:
print('Failed: %s: %s' % (type(e).__name__, e))
• Error handling with callbacks
• Exception propagates out of fut.result() method
51. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
51
def run():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(result_handler)
def result_handler(fut):
try:
result = fut.result()
print('Got:', result)
except Exception as e:
print('Failed: %s: %s' % (type(e).__name__, e))
• Consider the structure of code using futures
• Meditate on it... focus on the code.
• This seems sort of familiar
52. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Callback Hell?
52
• No, no, no.... keep focusing.
53. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
53
def entry():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(exit)
def exit(fut):
try:
result = fut.result()
print('Got:', result)
except Exception as e:
print('Failed: %s: %s' % (type(e).__name__, e))
• What if the function names are changed?
• Wait! This is almost a context manager (yes)
54. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Futures
54
@inlined_future
def do_func(x, y):
result = yield pool.submit(func, x, y)
print('Got:', result)
run_inline_future(do_func)
• Thought: Maybe you could do that yield trick
• The extra callback function is eliminated
• Now, just one "simple" function
• Inspired by @contextmanager
55. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
DéjàVu
55
56. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
DéjàVu
56
• This twisted idea has been used before...
57. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Preview
57
t = Task(gen)
• There are two separate parts
• Part 1:Wrapping generators with a "task"
• Part 2: Implementing some runtime code
run_inline_future(gen)
• Forewarning: It will bend your mind a bit
58. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
58
• Will continue to use threads for examples
• Mainly because they're easy to work with
• And I don't want to get sucked into an event loop
• Don't dwell on it too much
• Key thing: There is some background processing
59. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
59
def do_func(x, y):
result = yield pool.submit(func, x, y)
print('Got:', result)
• Problem: Stepping through a generator
• Involves gluing callbacks and yields together
def do_func(x, y):
yield pool.submit(func, x, y)
result =
print('Got:', result)
add_done_callback()
cut
enter
exit
60. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
60
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
61. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
61
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Task class wraps
around and represents
a running generator.
62. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
62
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Advance the
generator to the next
yield, sending in a
value (if any)
63. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
63
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Attach a callback to
the produced Future
64. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
64
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Collect result and
send back into the
generator
65. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
65
pool = ThreadPoolExecutor(max_workers=8)
def func(x, y):
time.sleep(1)
return x + y
def do_func(x, y):
result = yield pool.submit(func, x, y)
print('Got:', result)
t = Task(do_func(2, 3))
t.step()
• Try it:
• Output:
Got: 5
• Yes, it works
Note: must initiate
first step of the task
to get it to run
66. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
66
pool = ThreadPoolExecutor(max_workers=8)
def func(x, y):
time.sleep(1)
return x + y
def do_many(n):
while n > 0:
result = yield pool.submit(func, n, n)
print('Got:', result)
n -= 1
t = Task(do_many(10))
t.step()
• More advanced: multiple yields/looping
• Yes, this works too.
67. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
67
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
try:
result = fut.result()
self.step(result, None)
except Exception as exc:
self.step(None, exc)
68. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
68
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
try:
result = fut.result()
self.step(result, None)
except Exception as exc:
self.step(None, exc)
send() or throw()
depending on
success
69. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
69
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
try:
result = fut.result()
self.step(result, None)
except Exception as exc:
self.step(None, exc)
Catch exceptions
and pass to next
step as appropriate
70. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Error Example
70
def do_func(x, y):
try:
result = yield pool.submit(func, x, y)
print('Got:', result)
except Exception as e:
print('Failed:', repr(e))
t = Task(do_func(2, 'Hello'))
t.step()
• Try it:
• Output:
Failed: TypeError("unsupported operand type(s) for +:
'int' and 'str'",)
• Yep, that works too.
71. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
71
• This whole thing is rather bizarre
• Execution of the inlined future takes place all on
its own (concurrently with other code)
• The normal rules don't apply
73. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 4
73
yield from yield from yield from yield from future
(maybe)
source: @UrsulaWJ
74. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
74
• Focus on the future
• Not the past
• Not now
• Yes, the future.
• No, really, the future.
(but not the singularity)
75. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
75
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
...
generator must only
produce Futures
76. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
76
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• It's trying to delay the execution of a user-
supplied inlined future until later.
77. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
77
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• No
Traceback (most recent call last):
...
AttributeError: 'generator' object has no attribute
'add_done_callback'
78. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
78
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• This is busted
• gen is a generator, not a Future
79. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
79
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
for f in gen:
yield f
Task(after(10, do_func(2, 3))).step()
• What about this?
• Idea: Just iterate the generator manually
• Make it produce the required Futures
80. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
80
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
for f in gen:
yield f
Task(after(10, do_func(2, 3))).step()
• What about this?
• No luck.The result gets lost somewhere
Got: None
• Hmmm.
81. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
81
def after(delay, gen):
yield pool.submit(time.sleep, delay)
result = None
try:
while True:
f = gen.send(result)
result = yield f
except StopIteration:
pass
Task(after(10, do_func(2, 3))).step()
• Obvious solution (duh!)
• Hey, it works!
Got: 5
82. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
82
def after(delay, gen):
yield pool.submit(time.sleep, delay)
result = None
try:
while True:
f = gen.send(result)
result = yield f
except StopIteration:
pass
Task(after(10, do_func(2, 3))).step()
• Obvious solution (duh!)
• Hey, it works!
Got: 5
manual running of
generator with
results (ugh!)
83. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (4th Attempt)
83
def after(delay, gen):
yield pool.submit(time.sleep, delay)
yield from gen
Task(after(10, do_func(2, 3))).step()
• A better solution: yield from
• 'yield from' - Runs the generator for you
Got: 5
• And it works! (yay!)
• Awesome
84. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
PEP 380
• yield from gen - Delegate to a subgenerator
84
def generator():
...
yield value
...
return result
def func():
result = yield from generator()
• Transfer control to other generators
• Operations take place at the current yield
• Far more powerful than you might think
85. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Conundrum
85
def after(delay, gen):
yield pool.submit(time.sleep, delay)
yield from gen
• "yield" and "yield from"?
• Two different yields in the same function
• Nobody will find that confusing (NOT!)
86. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
86
def after(delay, gen):
yield from pool.submit(time.sleep, delay)
yield from gen
Task(after(10, do_func(2, 3))).step()
• Maybe this will work?
• Just use 'yield from'- always!
87. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
87
def after(delay, gen):
yield from pool.submit(time.sleep, delay)
yield from gen
Task(after(10, do_func(2, 3))).step()
• Maybe this will work?
• Just use 'yield from'- always!
• No. 'yield' and 'yield from' not interchangeable:
Traceback (most recent call last):
...
TypeError: 'Future' object is not iterable
>>>
88. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
??????
88
(Can it be made to work?)
89. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Iterable Futures
89
def patch_future(cls):
def __iter__(self):
! if not self.done():
yield self
return self.result()
cls.__iter__ = __iter__
from concurrent.futures import Future
patch_future(Future)
• A simple ingenious patch
• It makes all Future instances iterable
• They simply produce themselves and the result
• It magically makes 'yield from' work!
90. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
All Roads Lead to Future
90
yield self
• Future is the only thing
that actually yields
• Everything else delegates
using 'yield from'
• Future terminates the
chain
generator
generator
generator
Future
yield from
yield from
yield from
run()
next(gen)
91. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Decorator
91
import inspect
def inlined_future(func):
assert inspect.isgeneratorfunction(func)
return func
• Generators yielding futures is its own world
• Probably a good idea to have some demarcation
• Does nothing much at all, but serves as syntax
@inlined_future
def after(delay, gen):
yield from pool.submit(time.sleep, delay)
yield from gen
• Alerts others about what you're doing
92. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Wrangling
92
t = Task(gen)
t.step()
• The "Task" object is just weird
• No way to obtain a result
• No way to join with it
• Or do much of anything useful at all
Task
runs
????
t.step()
93. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
93
class Task(Future):
def __init__(self, gen):
!super().__init__()
! self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
self.set_result(exc.value)
• This tiny tweak makes it much more interesting
94. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
94
class Task(Future):
def __init__(self, gen):
!super().__init__()
! self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
self.set_result(exc.value)
• This tiny tweak makes it much more interesting
A Task is a Future
Set its result upon
completion
95. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
95
@inlined_future
def do_func(x, y):
result = yield pool.submit(func, x, y)
return result
t = Task(do_func(2,3))
t.step()
...
print("Got:", t.result())
• Obtaining the result of task
• So, you create a task that runs a generator
producing Futures
• The task is also a Future
• Right. Got it.
96. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
96
@inlined_future
def do_func(x, y):
result = yield pool.submit(func, x, y)
return result
t = Task(do_func(2,3))
t.step()
...
print("Got:", t.result())
class Task(Future):
...
def step(self, value=None, exc=None):
try:
...
except StopIteration as exc:
self.set_result(exc.value)
...
97. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Runners
97
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
def run_inline_future(fut):
t = start_inline_future(fut)
return t.result()
• You can make utility functions to hide details
result = run_inline_future(do_func(2,3))
print('Got:', result)
• Example: Run an inline future to completion
• Example: Run inline futures in parallel
t1 = start_inline_future(do_func(2, 3))
t2 = start_inline_future(do_func(4, 5))
result1 = t1.result()
result2 = t2.result()
98. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Step Back Slowly
98
• Built a generator-based task system for threads
Tasks
@inline_future
run_inline_future()
Threads
pools
concurrent.future
Futures
submit
result
• Execution of the future hidden in background
• Note: that was on purpose (for now)
99. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
99
• Ideas are the foundation asyncio coroutines
Tasks
@coroutine
run_until_complete()
Event Loop
asyncio
Futures
result
• In fact, it's almost exactly the same
• Naturally, there are some details with event loop
100. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Simple Example
100
import asyncio
def func(x, y):
return x + y
@asyncio.coroutine
def do_func(x, y):
yield from asyncio.sleep(1)
return func(x, y)
loop = asyncio.get_event_loop()
result = loop.run_until_complete(do_func(2,3))
print("Got:", result)
• asyncio "hello world"
101. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
101
import asyncio
@asyncio.coroutine
def echo_client(reader, writer):
while True:
line = yield from reader.readline()
if not line:
break
resp = b'Got:' + line
writer.write(resp)
writer.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(
asyncio.start_server(echo_client, host='', port=25000)
)
loop.run_forever()
• asyncio - Echo Server
102. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Be on the Lookout!
102
104. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 5
104
"Gil"
105. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Python Threads
105
• Threads, what are they good for?
• Answer: Nothing, that's what!
• Damn you GIL!!
106. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actually...
106
• Threads are great at doing nothing!
time.sleep(2) # Do nothing for awhile
data = sock.recv(nbytes) # Wait around for data
data = f.read(nbytes)
• In fact, great for I/O!
• Mostly just sitting around
107. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
107
• Threads are weak for computation
• Global interpreter lock only allows 1 CPU
• Multiple CPU-bound threads fight each other
• Could be better
http://www.dabeaz.com/GIL
108. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Solution
108
• Naturally, we must reinvent the one thing that
threads are good at
• Namely, waiting around.
• Event-loops, async, coroutines, green threads.
• Think about it: These are focused on I/O
(yes, I know there are other potential issues with threads, but work with me here)
109. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
109
• Event-loops have their own issues
• Don't bug me, I'm blocking right now
(source: chicagotribune.com)
110. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Standard Solution
110
• Delegate the work out to a process pool
python
python
python
python
python
main program
CPU
CPU
CPU
CPU
• multiprocessing, concurrent.futures, etc.
111. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
111
• Didn't we just do this with inlined futures?
def fib(n):
return 1 if n <= 2 else (fib(n-1) + fib(n-2))
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
val = yield from pool.submit(fib, i)
result.append(val)
return result
pool = ProcessPoolExecutor(4)
result = run_inline_future(compute_fibs(35))
• It runs without crashing (let's ship it!)
112. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
112
• Can you launch tasks in parallel?
t1 = start_inline_future(compute_fibs(34))
t2 = start_inline_future(compute_fibs(34))
result1 = t1.result()
result2 = t2.result()
• Sequential execution
run_inline_future(compute_fibs(34))
run_inline_future(compute_fibs(34))
• Recall (from earlier)
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
113. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
113
• Can you launch tasks in parallel?
t1 = start_inline_future(compute_fibs(34))
t2 = start_inline_future(compute_fibs(34))
result1 = t1.result()
result2 = t2.result()
• Sequential execution
run_inline_future(compute_fibs(34))
run_inline_future(compute_fibs(34))
• Recall (from earlier)
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
9.56s
114. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
114
• Can you launch tasks in parallel?
t1 = start_inline_future(compute_fibs(34))
t2 = start_inline_future(compute_fibs(34))
result1 = t1.result()
result2 = t2.result()
• Sequential execution
run_inline_future(compute_fibs(34))
run_inline_future(compute_fibs(34)) 9.56s
• Recall (from earlier)
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
4.78s
Inlined tasks running
outside confines of
the GIL?
115. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Execution Model
115
• The way in which it works is a little odd
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
print(threading.current_thread())
val = yield from pool.submit(fib, i)
result.append(val)
return result
add
this
• Output: (2 Tasks)
<_MainThread(MainThread, started 140735086636224)>
<_MainThread(MainThread, started 140735086636224)>
<Thread(Thread-1, started daemon 4320137216)>
<Thread(Thread-1, started daemon 4320137216)>
<Thread(Thread-1, started daemon 4320137216)>
...
????
116. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Process Pools
116
• Process pools involve a hidden result thread
main thread python
python
CPU
CPU
submit
result_thread
• result thread reads returned values
• Sets the result on the associated Future
• Triggers the callback function (if any)
results
117. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Issue
117
• Our inlined future switches execution threads
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
val = yield from pool.submit(fib, i)
result.append(val)
return result
main thread
result thread
• Switch occurs at the first yield
• All future execution occurs in result thread
• That could be a little weird (especially if it blocked)
118. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Important Lesson
118
• If you're going to play with control flow, you
must absolutely understand possible implications
under the covers (i.e., switching threads across
the yield statement).
119. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Insight
119
• The yield is not implementation
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
val = yield from pool.submit(fib, i)
result.append(val)
return result
• You can implement different execution models
• You don't have to follow a formulaic rule
120. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Thread Execution
120
• Variant: Run generator entirely in a single thread
def run_inline_thread(gen):
value = None
exc = None
while True:
try:
if exc:
fut = gen.throw(exc)
else:
fut = gen.send(value)
try:
value = fut.result()
exc = None
except Exception as e:
exc = e
except StopIteration as exc:
return exc.value
• It just steps through... no callback function
121. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
121
• Try it again with a thread pool (because why not?)
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
print(threading.current_thread())
val = yield from pool.submit(fib, i)
result.append(val)
return result
tpool = ThreadPoolExecutor(8)
t1 = tpool.submit(run_inline_thread(compute_fibs(34)))
t2 = tpool.submit(run_inline_thread(compute_fibs(34)))
result1 = t1.result()
result2 = t2.result()
122. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
122
• Output: (2 Threads)
<Thread(Thread-1, started 4319916032)>
<Thread(Thread-2, started 4326428672)>
<Thread(Thread-1, started 4319916032)>
<Thread(Thread-2, started 4326428672)>
<Thread(Thread-1, started 4319916032)>
<Thread(Thread-2, started 4326428672)>
...
(works perfectly)
4.60s
(a bit faster)
• Processes, threads, and futures in perfect harmony
• Uh... let's move along. Faster. Must go faster.
123. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Big Idea
123
• You can mold and adapt generator execution
• That yield statement: magic!
124. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 6
124
Fake it until you make it
125. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actors
125
• There is a striking similarity between coroutines
and actors (i.e., the "actor" model)
• Features of Actors
• Receive messages
• Send messages to other actors
• Create new actors
• No shared state (messages only)
• Can coroutines serve as actors?
126. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
126
• A very simple example
@actor
def printer():
while True:
msg = yield
print('printer:', msg)
printer()
n = 10
while n > 0:
send('printer', n)
n -=1
idea: use generators
to define a kind of
"named" actor task
127. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Attempt 1
127
_registry = { }
def send(name, msg):
_registry[name].send(msg)
def actor(func):
def wrapper(*args, **kwargs):
gen = func(*args, **kwargs)
next(gen)
_registry[func.__name__] = gen
return wrapper
• Make a central coroutine registry and a decorator
• Let's see if it works...
128. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
128
• A very simple example
@actor
def printer():
while True:
msg = yield
print('printer:', msg)
printer()
n = 10
while n > 0:
send('printer', n)
n -=1
• It seems to work (maybe)
printer: 10
printer: 9
printer: 8
...
printer: 1
129. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
129
• Recursive ping-pong (inspired by Stackless)
@actor
def ping():
while True:
n = yield
print('ping %d' % n)
send('pong', n + 1)
@actor
def pong():
while True:
n = yield
print('pong %d' % n)
send('ping', n + 1)
ping()
pong()
send('ping', 0)
ping
pong
130. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
130
ping 0
pong 1
Traceback (most recent call last):
File "actor.py", line 36, in <module>
send('ping', 0)
File "actor.py", line 8, in send
_registry[name].send(msg)
File "actor.py", line 24, in ping
send('pong', n + 1)
File "actor.py", line 8, in send
_registry[name].send(msg)
File "actor.py", line 31, in pong
send('ping', n + 1)
File "actor.py", line 8, in send
_registry[name].send(msg)
ValueError: generator already executing
• Alas, it does not work
131. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Problems
131
• Important differences between actors/coroutines
• Concurrent execution
• Asynchronous message delivery
• Although coroutines have a "send()", it's a normal
method call
• Synchronous
• Involves the call-stack
• Does not allow recursion/reentrancy
132. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 1
132
class Actor(threading.Thread):
def __init__(self, gen):
super().__init__()
self.daemon = True
self.gen = gen
self.mailbox = Queue()
self.start()
def send(self, msg):
self.mailbox.put(msg)
def run(self):
while True:
msg = self.mailbox.get()
self.gen.send(msg)
• Wrap the generator with a thread
• Err...... no.
133. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 2
133
_registry = { }
_msg_queue = deque()
def send(name, msg):
_msg_queue.append((name, msg))
def run():
while _msg_queue:
name, msg = _msg_queue.popleft()
_registry[name].send(msg)
• Write a tiny message scheduler
• send() simply drops messages on a queue
• run() executes as long as there are messages
134. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
134
• Recursive ping-pong (reprise)
@actor
def ping():
while True:
n = yield
print('ping %d' % n)
send('pong', n + 1)
@actor
def pong():
while True:
n = yield
print('pong %d' % n)
send('ping', n + 1)
ping()
pong()
send('ping', 0)
run()
ping
pong
135. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
135
ping 0
pong 1
ping 2
pong 3
ping 4
ping 5
ping 6
pong 7
...
... forever
• It works!
• That's kind of amazing
136. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Comments
136
• It's still kind of a fake actor
• Lacking in true concurrency
• Easily blocked
• Maybe it's good enough?
• I don't know
• Key idea: you can bend space-time with yield
137. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 7
137
A TerrifyingVisitor
138. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Let's Write a Compiler
138
• Well, an extremely simple one anyways...
• Evaluating mathematical expressions
2 + 3 * 4 - 5
• Why?
• Because eval() is for the weak, that's why
139. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
139
• Lexing : Make tokens
2 + 3 * 4 - 5 [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
• Parsing : Make a parse tree
[NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
PLUS
NUM
(2)
TIMES
NUM
(3)
NUM
(4)
MINUS
NUM
(5)
140. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
140
• Evaluation :Walk the parse tree
PLUS
NUM
(2)
TIMES
NUM
(3)
NUM
(4)
MINUS
NUM
(5)
9
• It's almost too simple
141. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
141
import re
from collections import namedtuple
tokens = [
r'(?P<NUM>d+)',
r'(?P<PLUS>+)',
r'(?P<MINUS>-)',
r'(?P<TIMES>*)',
r'(?P<DIVIDE>/)',
r'(?P<WS>s+)',
]
master_re = re.compile('|'.join(tokens))
Token = namedtuple('Token', ['type','value'])
def tokenize(text):
scan = master_re.scanner(text)
return (Token(m.lastgroup, m.group())
for m in iter(scan.match, None)
if m.lastgroup != 'WS')
142. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
142
text = '2 + 3 * 4 - 5'
for tok in tokenize(text):
print(tok)
Token(type='NUM', value='2')
Token(type='PLUS', value='+')
Token(type='NUM', value='3')
Token(type='TIMES', value='*')
Token(type='NUM', value='4')
Token(type='MINUS', value='-')
Token(type='NUM', value='5')
• Example:
143. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Parsing
143
• Must match the token stream against a grammar
expr ::= term { +|- term }*
term ::= factor { *|/ factor}*
factor ::= NUM
• An expression is just a bunch of terms
2 + 3 * 4 - 5
term + term - term
• A term is just one or more factors
term term
factor factor * factor
(2) (3) (4)
144. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
144
expr ::= term { +|- term }*
term ::= factor { *|/ factor}*
factor ::= NUM
def expr():
! term()
! while accept('PLUS','MINUS'):
term()
!print('Matched expr')
def term():
! factor()
while accept('TIMES','DIVIDE'):
factor()
print('Matched term')
def factor():
if accept('NUM'):
print('Matched factor')
else:
raise SyntaxError()
Encode the grammar
as a collection of
functions
Each function steps
through the rule
145. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
145
def parse(toks):
lookahead, current = next(toks, None), None
def accept(*toktypes):
nonlocal lookahead, current
if lookahead and lookahead.type in toktypes:
current, lookahead = lookahead, next(toks, None)
return True
def expr():
term()
while accept('PLUS','MINUS'):
term()
print('Matched expr')
...
expr()
146. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
146
class Node:
_fields = []
def __init__(self, *args):
for name, value in zip(self._fields, args):
setattr(self, name, value)
class BinOp(Node):
_fields = ['op', 'left', 'right']
class Number(Node):
_fields = ['value']
• Need some tree nodes for different things
• Example:
n1 = Number(3)
n2 = Number(4)
n3 = BinOp('*', n1, n2)
147. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
147
def parse(toks):
...
def expr():
left = term()
while accept('PLUS','MINUS'):
left = BinOp(current.value, left)
left.right = term()
return left
def term():
left = factor()
while accept('TIMES','DIVIDE'):
left = BinOp(current.value, left)
left.right = factor()
return left
def factor():
if accept('NUM'):
return Number(int(current.value))
else:
raise SyntaxError()
return expr()
Building nodes
and hooking
them together
148. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Our Little Parser
148
text = '2 + 3*4 - 5'
toks = tokenize(text)
tree = parse(toks)
• Story so far...
BinOp('-',
BinOp('+',
Number(2),
BinOp('*',
Number(3),
Number(4)
)
),
Number(5)
)
151. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Digression
151
• Last 12 slides a whole graduate CS course
• Plus at least one additional Python tutorial
• Don't worry about it
• Left as an exercise...
152. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Death Spiral
152
# Make '0+1+2+3+4+...+999'
text = '+'.join(str(x) for x in range(1000))
toks = tokenize(text)
tree = parse(toks)
val = Evaluate().visit(tree)
• And it almost works...
Traceback (most recent call last):
File "compiler.py", line 100, in <module>
val = Evaluator().visit(tree)
File "compiler.py", line 63, in visit
return getattr(self, 'visit_' + type(node).__name__)(node)
File "compiler.py", line 80, in visit_BinOp
leftval = self.visit(node.left)
...
RuntimeError: maximum recursion depth exceeded while calling a
Python object
155. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 155
• The visitor pattern is bad idea
• Better: Functional language with pattern
matching and tail-call optimization
I ToldYou So
156. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
156
How do you NOT do something?
157. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
157
How do you NOT do something?
(yield?)
159. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
159
class NodeVisitor:
def genvisit(self, node):
result = getattr(self,
'visit_' + type(node).__name__)(node)
if isinstance(result, types.GeneratorType):
result = yield from result
return result
• Step 1:Wrap "visiting" with a generator
• Thinking: No matter what the visit_() method
produces, the result will be a generator
• If already a generator, then just delegate to it
160. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
160
>>> v = Evaluator()
>>> n = Number(2)
>>> gen = v.genvisit(n)
>>> gen
<generator object genvisit at 0x10070ab88>
>>> gen.send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 2
>>>
• Example:A method that simply returns a value
• Result: Carried as value in StopIteration
161. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
161
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> gen = v.genvisit(n)
>>> gen
<generator object genvisit at 0x10070ab88>
>>> gen.send(None)
<__main__.Number object at 0x1058525c0>
>>> gen.send(_.value)
<__main__.Number object at 0x105852630>
>>> gen.send(_.value)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
• A method that yields nodes (iteration)
Again, note the return
mechanism
162. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
162
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> gen = v.genvisit(n)
>>> gen
<generator object genvisit at 0x10070ab88>
>>> gen.send(None)
<__main__.Number object at 0x1058525c0>
>>> gen.send(_.value)
<__main__.Number object at 0x105852630>
>>> gen.send(_.value)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
• A method that yields nodes
def visit_Number(self, node):
return node.value
Manually carrying out this
method in the example
163. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
163
class NodeVisitor:
def visit(self, node):
stack = [ self.genvisit(node) ]
! result = None
while stack:
try:
node = stack[-1].send(result)
stack.append(self.genvisit(node))
result = None
except StopIteration as exc:
! stack.pop()
result = exc.value
! return result
• Step 2: Run depth-first traversal with a stack
• Basically, a stack of running generators
164. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Transcendence
164
# Make '0+1+2+3+4+...+999'
text = '+'.join(str(x) for x in range(1000))
toks = tokenize(text)
tree = parse(toks)
val = Evaluate().visit(tree)
print(val)
• Does it work?
• Yep
499500
• Yow!
165. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
165
class Evaluator(NodeVisitor):
def visit_BinOp(self, node):
leftval = yield node.left
rightval = yield node.right
if node.op == '+':
result = leftval + rightval
...
return result
• Each yield creates a new stack entry
• Returned values (via StopIteration) get
propagated as results
generator
generator
generator
stack
yield
yield return
return
166. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
166
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> stack = [ v.genvisit(n) ]
>>> stack[-1].send(None)
<__main__.Number object at 0x1058525c0>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 3
>>> stack.pop()
>>> stack[-1].send(3)
<__main__.Number object at 0x105852630>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 4
>>> stack.pop()
>>> stack[-1].send(4)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
Nodes are visited and
generators pushed onto
a stack
167. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
167
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> stack = [ v.genvisit(n) ]
>>> stack[-1].send(None)
<__main__.Number object at 0x1058525c0>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 3
>>> stack.pop()
>>> stack[-1].send(3)
<__main__.Number object at 0x105852630>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 4
>>> stack.pop()
>>> stack[-1].send(4)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
Results propagate via
StopIteration
12 (Final Result)
168. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 168
Final Words
169. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Historical Perspective
169
• Generators seem to have started as a simple
way to implement iteration (Python 2.3)
• Took an interesting turn with support for
coroutines (Python 2.5)
• Taken to a whole new level with delegation
support in PEP-380 (Python 3.3).
170. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Control Flow Bending
170
• yield statement allows you to bend control-flow
to adapt it to certain kinds of problems
• Wrappers (context managers)
• Futures/concurrency
• Messaging
• Recursion
• Frankly, it blows my mind.
171. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
171
• Inclusion of asyncio in standard library may be a
game changer
• To my knowledge, it's the only standard library
module that uses coroutines/generator
delegation in a significant manner
• To really understand how it works, need to have
your head wrapped around generators
• Read the source for deep insight
172. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Is it Proper?
172
• Are coroutines/generators a good idea or not?
• Answer: I still don't know
• Issue: Coroutines seem like they're "all in"
• Fraught with potential mind-bending issues
• Example:Will there be two standard libraries?
173. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Two Libraries?
173
Python
Standard Library
Standard
coroutine library
(asyncio and friends)
?????
• If two different worlds, do they interact?
• If so, by what rules?
174. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Personal Use
174
• My own code is dreadfully boring
• Generators for iteration: Yes.
• Everything else: Threads, recursion, etc. (sorry)
• Nevertheless:There may be something to all of
this advanced coroutine/generator business
175. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Bit More Information
175
176. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thanks!
176
• I hope you got some new ideas
• Please feel free to contact me
http://www.dabeaz.com
• Also, I teach Python classes (shameless plug)
@dabeaz (Twitter)
• Special Thanks:
Brian Curtin, Ken Izzo, George Kappel, Christian Long,
Michael Prentiss,Vladimir Urazov, Guido van Rossum