Features new to Python 3.6 (plus a brief discussion of async features across languages)

1. Python 3.6 Features
and a Cross-Language Comparison of Async Features
2016-12-07
Python 3.6 development began: 2016-05-24 …
… final release expected: 2016-12-16
1

2. Brief History of Python releases
1994 – Python 1.0, 1.1
1995 – Python 1.2, 1.3
1996 – Python 1.4
1998 – Python 1.5
2000 – Python 1.6
2001 – Python 2.1, 2.2
2003 – Python 2.3
2004 – Python 2.4
2006 – Python 2.5
2008 – Python 2.6 (Oct 1)
2010 – Python 2.7
2014 – “2.7 supported until 2020”
2008 – Python 3.0 (Dec 3)
2009 – Python 3.1
2011 – Python 3.2
2012 – Python 3.3
2014 – Python 3.4
2015 – Python 3.5
2016 – Python 3.6
• v3.0: thread no more;
use threading instead
• 2010: Celery 1.0.0 released
• v3.4: asyncio added (provisional)
2

3. Python 3.6 Feature PEPs
(as listed in PEP 494; ?? = “Provisional”)
Less interesting 3.6 PEPs:
• {PEP 468, PEP 520 }: Preserving Order for {Keyword Argument, Class Attribute Definition}
• PEP 495: Local Time Disambiguation
• PEP 498: Literal String Formatting
• PEP 506: Adding A Secrets Module To The Standard Library
• PEP 509: Add a private version to dict
• PEP 515: Underscores in Numeric Literals
• PEP 519: Adding a file system path protocol
• {PEP 528, PEP 529}: Change Windows encoding to UTF-8 for {console,filesystem} (??)
More interesting 3.6 PEPs!
• PEP 487! Simpler customization of class creation
• PEP 523! Adding a frame evaluation API to CPython
• PEP 524! Make os.urandom() blocking on Linux (during system startup)
• PEP 526! Syntax for Variable Annotations (??)
Most interesting—Async!!
• PEP 525!! Asynchronous Generators (??)
• PEP 530!! Asynchronous Comprehensions
3

4. PEP 468: Preserving Order for Keyword Arguments
Review
• Calling. Python can assign argument values by position or by keyword.
(not to be confused with required vs. optional arguments)
So f(x, y, z) can be called like so:
• by position: f(1, 2, 3) ~or~ vec = [1,2,3]; f(*vec) # starred syntax
• by keyword: f(x=1, y=2, z=3) ~or~ d = {'x':1, 'y':2, 'z':3}; f(**d)
• Defining. Function defs can “slurp” in ordered args or keyword args.
The Problem. Losing potentially valuable information; Serialization is inconsistent.
The Change. Keyword ordering will been preserved by being passed as an OrderedDict.
• Performance was at one point a concern, but there is now a C impl. of OrderedDict.
Ordered arguments unpacking syntax: Keyword arguments unpacking syntax
def print_args(*pargs):
for a in pargs:
print(a)
def print_named_args(**kwargs):
for name, arg in kwargs:
print('Arg {0} = {1!s}'.format(name, arg))
4
Fun fact: You can also
use starred syntax in
class base lists when
defining classes.

5. PEP 520: Preserving Order for Class Attribute Definitions
• PEP 520 is similar to PEP 468, but applies to class attributes (i.e., members, fields).
• The class definition namespace is now implemented w/ OrderedDict instead of a dict, though:
• “Dunder” attributes (e.g., __init__, __module__) are ignored (b/c reserved for the interpreter.)
• The long-lived class namespace __dict__ will continue to use a dict.
• This *could* be done before, by defining __prepare__() in a metaclass (see PEP 3115),
but metaclasses are trippy, and most people would prefer not to use them.
• __definition_order__ is a new read-only tuple, will record the attribute definition order, allowing
introspection, e.g., by class decorators.
5

6. PEP 495: Local Time Disambiguation
6
Problem. "I know it's 1:50am—I just don't know which 1:50am."
• "fold"—What happens to time when the clocks are turned back. Ambiguity results!
• "gap"—What happens to time when the clocks are turned forward.
Change. Add a new attribute "fold" to datetime.time and datetime.datetime.
• fold == 1 only when a moment in time is in the second occurrence of a day’s time.
• fold == 0 at all others times.
>>> datetime(2016, 11, 6, 1, 30, fold=0).timestamp()
1478424600.0
>>> datetime(2016, 11, 6, 1, 30, fold=1).timestamp()
1478428200.0
• __new__() and replace() methods will get a keyword-only argument called fold=0.
• Serialization ("pickling") of (date)times will use a currently unused bit.
For backwards compatibility, utcoffset(), tzname(), and dst() will be unchanged.

7. PEP 498: Literal String Formatting
• Python supports a few ways to format strings: %, str.format(), and string.Template.
• Enter a fourth way: Literal String Interpolation. We’ll call such strings "f-strings".
• They "provide a concise, readable way to include the value of expressions inside strings".
• Example from the PEP (modified somewhat):
>>> import datetime
>>> (name, age, anniversary) = ('Fred', 50, datetime.date(1991, 10, 12))
>>> f'My name is {name}; age next year is {age+1}; anniversary is {anniversary:%A, %B %d, %Y}.'
'My name is Fred; age next year is 51; anniversary is Saturday, October 12, 1991.'
• Allowed: f'{{ {4 * 10} }}' # Doubling braces preserves them in output; Result = '{ 40 }'
• Allowed: fr'x={4*10}n' # Formatted and raw string qualifiers can be used together.
• Allowed: f'{"quoted string"}' # Different types of quotes are OK.
• Not allowed: f'{'quoted string'}' # Backslashes forbidden in interpolated expression.
• The PEP describes many other valid and invalid use cases.
7

8. PEP 506: Adding A Secrets Module To The Standard Library
• Motivation: Separate cryptographically strong random from regular random,
so that developers will be less likely to write unsafe code.
• Note: The libraries cryptography and PyCrypto already exist.
• Main changes:
• To generate random tokens, secrets.token_bytes(), secrets.token_hex(), etc.
• To choose items from a sequence, secrets.choice()
• To generate random bits/bytes as an integer, secrets.randbits()
• To generate a random integer in a half-open range, secrets.randbelow()
8

9. PEP 509: Add a private version (number) to dict
• From the PEP: "Python is hard to optimize because almost everything is
mutable…".
• Dicts are used to represent namespaces, including the global namespace. Put
simply, data in namespace caches can be used until the namespace is
modified, at which point the cache must be partially or completely flushed to
avoid using stale data.
• This PEP introduces a version number for each dict, incremented by one with
every change. These version numbers acts as "guards" against using stale data.
• Version numbers can be read via the function dict_get_version(dict).
• Version overflow will falsely report a lack of changes once every 2**64 times.
9

10. PEP 515: Underscores in Numeric Literals
• Allow:
• amount = 10_000_000.0
• addr = 0xCAFE_F00D
• flags = 0b_0011_1111_0100_1110
• flags = int('0b_1111_0000', 2)
Also:
• The PEP contains revised EBNF for Python numeric literals,
and a list of other languages with a similar feature.
• If an underscore is used as a separator for b-, x-, and o-formatted strings,
then the digits will be put into groups of 4.
10

11. PEP 519: Adding a file system path protocol
• This extends Path objects (introduced in PEP 428; Py 3.4) through the std lib.
p = pathlib.PurePath('/home/antoine/Pathlib/setup.py')
// Later on, pass str(p) to a library function.
• Can we just call str() on any path-related object to get a string representation?
• No—the string representation of dirent=os.DirEntry is str(dirent.path)
• So, to be uniform without changing DirEntry, we need a protocol for paths.
• Any "path-like" object will define the method __fspath__()
• This method will also be added to DirEntry.
• The new function os.fspath(path) will return path if it’s a string or bytes.
Otherwise, it will return type(path).__fspath__(path)
11

12. PEP 528 & PEP 529: Change Windows encoding to UTF-8
for console & filesystem
• Problem: When Python is run in a Windows console (or filesystem), some
characters are displayed as garbage.
• Solution: Join the 21st century, and use UTF-16 UTF-8.
• Backwards compatibility: You can set the environment variable
PYTHONLEGACYWINDOWSSTDIO ifyouwanttoretaintheoldbehavior.
12

13. ¡ More interesting 3.6 PEPs !
13

14. PEP 487! Simpler customization of class creation
• Did you know that in Python you can define a class at runtime?
MyClass = type('MyClass', [ … ], { … }) # 2nd arg=base classes; 3rd arg=methods
my_instance = MyClass()
So type is a factory of classes, just like MyClass is a factory of MyClass instances.
• Everything in Python—including a class—is an object created by its "metaclass".
(except type)
• E.g.: You can use a metaclass to auto-create web services for classes created.
• But defining your own metaclasses is a drag. Can’t we make this easier? Yes!
• This PEP adds hooks into the ordinary (metaclass-free) class creation process.
• __init_subclass__(): a method that customizes subclasses of the current one.
• __set_name__(): a method to make the name of the current class available to attributes.
14

15. PEP 523! Adding a frame evaluation API to CPython
• Allow a per-interpreter function pointer to handle the evaluation of frame.
• Example use cases: Adding a JIT to CPython; customized debugging
• Also: Add extra data (co_extra; NULL by default) to each code object, to be
used by the frame evaluation function via C APIs (…SetExtra, …GetExtra).
The other fields of code objects would remain immutable.
15

16. PEP 524! Make os.urandom() blocking on Linux [3.17+]
(during system startup)
• A new getrandom() syscall was added to Linux 3.17 and Python 3.5.0.
• Python started blocking on startup in Linux VMs & embedded devices, while the
kernel initialized urandom w/ 128 bits of entropy. This can take minutes.
• This is a significant performance problem in cases where a Python script uses
randomness early in the init process, e.g., to avoid predictability. In some cases,
this causes the init process to never complete!!!
• Changes:
• Crypto-secure use case: Use os.urandom(); have that block until /dev/urandom is ready.
• Non-secure use case: Use new function os.getrandom(), which reads from /dev/random.
• /dev/random is unlikely to block.
16

17. PEP 526! Syntax for Variable Annotations
• PEP 3107 allowed for (unused) function type annotations.
• PEP 484 standardizes optional type annotation syntax, to be used for static analysis (type
checking, codegen, refactoring, etc.) These annotations can be obtained via the function
typing.get_type_hints().
• PEP 526 extends where type annotations can be applied to "any valid single [local]
assignment target", but not to arbitrary expressions.
• Examples:
c = MyClass()
c.x: int = 0 # Annotates c.x with int.
c.y: int # Annotates c.y with int.
d = {}
d['a']: int = 0 # Annotates d['a'] with int.
d['b']: int # Annotates d['b'] with int.
(x): int # Annotates x with int, (x) treated as expression by compiler.
(y): int = 0 # Same situation here.
17

18. ¡¡ Most interesting—Async !!
18

19. Pre-async digression: for loops
for item in stuff:
do_something(item)
19
Method #2 (Legacy): stuff has a __getitem__() method.
The string class uses this approach.
Method #1: stuff has an __iter__() method, which returns an iterator.
The iterator defines a __next__() method [was next() in Python 2].
When the iterator has run out of items, it raises an IndexError.
Note: Iterators also have an identity __iter__() method:
def __iter__(self): return self
This makes it always safe to call iter() on an iterable,
in order to obtain an iterator.
stuff is referred to as an iterable.
How is this implemented?
Execution:
it = stuff.__iter__()
item1 = it.__next__()
item2 = it.__next__()
item3 = it.__next__() # IndexError raised, and loop exits.
Note:
• Adding use of a yield expression turns an iterator into a generator (from Python 2.5).
• Adding send(), throw(), close() extends generators into coroutines (also Python 2.5; see PEP 342).

20. Pre-async digression: with statements (PEP 343)
with open(foo.txt) as f:
data = f.read()
do_something(data)
20
Exit: When execution of the block is completed (by
whatever means), f’s __exit__() function is called. This
frees up any file resources previously allocated.
"with" statements can be used for values that are context managers.
This means that they define __enter__() and __exit__() members.
Note: Defining __exit__() involves correctly handling exceptions.
Enter: When execution of the block begins, f’s __enter__()
function is called. This allocates file resources.
How are the (file) resources
allocated then later freed?
Execution:
f.__enter__() # Acquire file resources
data = f.read()
do_something(data)
f.__exit__() # Free file resources

21. Async introduction
• Async functions can suspend and resume their execution.
• There are async analogues to the special functions mentioned above.
21
Classic Async (see PEP 3156)
def defines a function/routine. async def defines a coroutine (PEP 492) .
__iter__, __next__ used by for __aiter__, __anext__, used by async for.
__enter__, __exit__ used by with __aenter__, __aexit__, used by awith,
each return an "awaitable".
Awaitables generally define __await__(),
which returns an iterator.
Classic Async (see PEP 3156)
def func(): # a function
return 42
async def coro(): # a coroutine function
return await something() # something is async!
def genfunc(): # a generator function
yield 41
yield 42
async def asyncgen(): # an async generator
yield await something()
yield 42

22. PEP 525!! Asynchronous Generators
• PEP 255 introduced Simple Generators—the "yield" statement (*) —to 2.2.
• PEP 492 introduced support for native coroutines async/await to 3.5.
• This PEP extends generators to the async use case:
22
(*) Note: yield was changed from a statement to an expression in Python 2.5 (2006).
Old New
class Ticker:
'''Yield #s from 0 to upper every delay secs.'''
def __init__(self, delay, upper):
self.delay = delay
self.i = 0
self.upper = upper
def __aiter__(self):
return self
async def ticker(delay, upper):
'''Yield #s from 0 to upper every delay secs.'''
for i in range(upper):
yield i
await asyncio.sleep(delay)async def __anext__(self):
i = self.i
if i >= self.to:
raise StopAsyncIteration
self.i += 1
if i:
await asyncio.sleep(self.delay)
return i

23. PEP 530!! Asynchronous Comprehensions
(for lists, sets, and dicts; only list examples shown)
• Async comprehensions are only allowed inside of an async def function.
23
Old New
def foo():
…
result = [ ]
async for i in aiter():
if i % 2:
result.append(i)
async def foo():
…
result = [ i for i in aiter() if i % 2 ]
def bar():
…
result = [ ]
for fun in funcs:
result.append(await fun())
async def bar():
…
result = [ await fun() for fun in funcs ]

24. ¡¡¡ Fascinating—Async across languages !!!
24

25. Async: Cross-language commonalities
• Concurrency vs. Parallelism vs. Multi-threading • Using different threads/processes/computers
• Shared memory vs. Message passing
• Async issues:
• Running tasks (threads, pools, futures, etc.)
• Task dependencies, priorities, etc. (using locks)
• Handling signals, exceptions, and timeouts
• Coroutines: State machine use cases
• Task cancellation
• Logging
25
Is your task CPU
bound?
Yes:
Abandon
CPU!!
No: OK –
Stay on
this CPU
• Same process
• Use threads
• Share state
• Separate processes
• Don’t share as much state
• Communicate via sockets
& message passing
Single-
threaded
Parallel?
N Y
Con-
current
?
N N/A
Y N/A
Multi-
threaded
Parallel?
N Y
Con-
current
?
N
Y
T1 T2 T3
T1 T2 T1 T2
T1 T2 T3
T1 T4 T3
T1 T2 T1
T1 T1 T3
T1 T2 T3
T4 T5 T6
T1 T2 T1
T4 T5 T4
See asyncio library
See both asyncio and multiprocessing libraries

26. Async in C#
C# got async/await keywords in v5.0 (2012)
async Task<int> webPageLengthAsync()
{
HttpClient client = new HttpClient();
Task<string> getStringTask =
client.GetStringAsync("http://bitcoins4u.com");
…
string urlContents = await getStringTask;
return urlContents.Length;
}
26

27. Async in Python
Python got async/await keywords v3.5 (2015)
import asyncio
async def slow_operation(n):
await asyncio.sleep(1)
print(f"Slow call #{n} complete")
async def main():
await asyncio.wait([
slow_operation(1),
slow_operation(2),
slow_operation(3),
])
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
27
Note: Older async functions that are not defined with
the async keyword should use the decorator
@asyncio.coroutine
• This enables the generator to use yield from to call
async def coroutines, and also to be called by async
def coroutines, e.g., in an await expression
• Such functions that exit without yielding a generator
will log a warning if debug mode is turned on in
asyncio by setting PYTHONASYNCIODEBUG to
1, or by calling AbstractEventLoop.set_debug().
There is no need to decorate async def functions.

28. Async in Java
Java and C++ use classes instead of syntactic sugar (async/await).
• Java 1.5 added Future (an interface) and FutureTask (impl of Future & Runnable).
Runnable task = () -> {
System.out.println("Hello, world!");
};
task.run();
~or~
Thread t = new Thread(task).start(); // fn could also be a Callable.
~or~
ExecutorService threadpool = Executors.newFixedThreadPool(3);
Future future = threadpool.submit(task);
System.out.print("Task is not completed yet")
while (! future.isDone()) {
System.out.print(".");
Thread.sleep(1000); // Sleep for 1 second
}
System.out.println("");
int result = future.get();
threadpool.shutdown(); 28
Note: Java 8 introduced the CompletableFuture,
which is a promise. It implements both Future<T>
and CompletionStage<T>. Other Futures include:
• ForkJoinTask
• FutureTask
• RecursiveAction
• RecursiveTask
• SwingWorker

29. Async in C++
C++ got the class std::async class in C++11. It doesn’t use async/await keywords.
// Compile with –std=c++11 –pthread
#include <chrono>
#include <future>
#include <iostream>
#include <thread>
void f() {
std::cout << "a";
// BTW, std::this_thread has a member called yield()
std::this_thread::sleep_for(std::chrono::seconds(1));
std::cout << "b";
}
void g() {
std::cout << "c";
std::this_thread::sleep_for(std::chrono::seconds(2));
std::cout << "d";
}
int main() {
std::async(std::launch::async, f), std::async(std::launch::async, g);
// TODO: Add join here on the preceding two Futures.
}
Output: acbd ~or~ cabd 29
Launch policies:
See also
std::launch::deferred
Note: C++11 has a many concurrency-related classes,
including:
• thread
• mutex (with timed, recursive, and shared
versions) and members lock() & try_lock()
• condition_variable
• future, promise

30. Async: Other cross-language differences
Before Python adopted async/await, there were 2+ optional available:
• "greenlets" in the gevent lib.
• In a greenlet, "target.switch(value) can take you to a specific target coroutine
and yield a value where the target would continue to run".
• Coroutines as defined today with async/await don’t support this feature.
• The very popular Twisted library preceded v3.5, and used yield instead of await.
30

31. 31
sys.exit(0)

32. Resources
• Callbacks as our Generations’ Go To Statement, by Miguel Icaza
• http://tirania.org/blog/archive/2013/Aug-15.html
• Cross-language discussion on async/await from 2014:
• https://news.ycombinator.com/item?id=8598064
• How and when to use async and await in C#
• http://stackoverflow.com/questions/14455293/how-and-when-to-use-async-and-await
• How the heck does async work in Python 3.5:
• http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5
• Java 8: Writing asynchronous code with CompletableFuture
• http://www.deadcoderising.com/java8-writing-asynchronous-code-with-completablefuture/
• Multithreading in C++11 (8 parts)
• https://www.justsoftwaresolutions.co.uk/threading/multithreading-in-c++0x-part-1-starting-
threads.html (with links to the other 7 parts)
32