Gentle introduction to asynchronous programming on .NET stack. Async-await construct of .Net languages (e.g. C#), its benefits, threads, thread-pool, task, asp.net request handling etc.
2. ASYNCHRONY
I Am Bit Tricky
To Catch
Trendy
Examples: AJAX, AMD etc. and numerous language-
level/addon implementations
Languages supporting async patterns:
C#:Tasks,
Java7-8: Future and CompletableFuture
Go: Routines and channels,
Erlang: Processes and spawn
Python: tasks, coroutines (asyncio)
JavaScript (ES2017-ES8): Promises
and list goes on.
3. ASYNCHRONY
I'll Call You Back
Occurrence of events independent of the main
program flow
Enables this “main program” continue its work, not
blocking to wait for event results
Implicit degree of parallelism
4. BENEFITS
I Am Not Just
For UI Apps
1. Scalability and
2. Offloading (e.g. responsiveness, parallelism)
Most client apps care about asynchrony for
“Offloading” reasons
For store, desktop and phone apps: primary benefit is
responsiveness
Most server apps care about asynchrony for
“Scalability” reasons
Node.js, ASP.NET Core are inherently asynchronous,
hence scaling is their key.
Common Question: Does async have a place on the
server?
5. GOAL
I Am About To
Discussed More?
History of Asynchronous APIs on .NET
Thread,Threadpool and Task
Conceptual overview of request handling on
ASP.NET
I won’t be covering C# async-await API syntax
6. HISTORY
.Net And .Net
Languages have
Always Been
Loving Async
APM (.NET1.1) : Socket class
Begin<MethodName> and End<MethodName>
methods
EAP (.NET2) : e.g. BackgroundWorker class
<MethodName>Async methods
Task and TPL (.NET4)
Task-based Asynchronous Pattern (TAP) (.NET4.5)
Async and await
Language level support on C# and VB (2012), F#
(2010)
TAP: Recommended asynchronous design pattern
7. THREAD
I Am An Action
Lady
First appears in IBM OS/360 in 1967
Its actual OS-level thread
Gives life to a process
Each process has at least one thread (Main thread)
Threads share their address space
8. CLR THREAD
You should hate
new Thread()
Freaking expensive: Memory and time overhead associated
with them
Thread =
(Thread Kernel Object) x86-700BYTE, x64-1240BYTE
+ (Thread Environment Block) x86-4KB, x64-8KB
+ (User Mode Stack) x86, x64-1MB
+ (Kernel Mode Stack) x86-12KB, x64-24KB
CLR thread directly maps to Windows thread
Highest degree of control that programmers don’t want
Only spun up new if heavy computations on multiple CPUs
needs to be done.
Foreground and background threads
9. THREADPOOL
I manage threads
for you, sir!
Thread pool is a set of threads available readily and
maintained by the CLR.
No control but pool size: Submit work to execute,
wait for good to happen
Best for large no. of short operations where the
caller does not need the result.
10. CONTINUED…
Threadpool size (no. of threads)
Default minimum = No. of processors
Default maximum = 5000 (.NET4.5)
Thread Injection: Starvation avoidance and hill-
climbing algorithm
Threads are added or removed every 500ms
Thread pool categorizes its threads as
Worker threads
I/O completion port (IOCP) threads
11. IIS: REQUEST
HANDLING
Web Dev? You
Need To
Understand Me
An app/web server
Kernel mode and user mode (Native mode)
App pool: Grouping of URLs that is routed to one or
more worker processes.
12. OLD SYNCHRONOUS
WAY
You Devs Love Me,
Don’t You?
ASP.NET takes one of its thread pool threads and
assigns it to just arrived request
Request handler call that external resource
synchronously and blocks it until result returns
13. BAD PART
All Busy,Try
Later
Thread count saturation
Available threads blocked and wasted
New request have to wait and in danger of 503
14. ASYNCHRONOUS
WAY
Don’t Trust Me?
Try Then
Async don’t waste precious threadpool threads
Server could cope new request easily
Smaller number of threads to handle a larger
number of requests.
15. THREADPOOL SIZE
Just Increase Me
And Forget Async
Altogether.
I Do Joke Too
Async does not replace the thread pool, rather
makes optimum use of it
Scales both further and faster than blocking
threadpool threads
Less memory pressure
Can respond to sudden swings in request volume
Common question: What About the Thread Doing
the AsynchronousWork? There must something
monitoring at it, right?
No, not at all
16. TASKS
Some Call Me
FUTURES Others
PROMISES
Best of both worlds
TaskScheduler
Thread Pool Task Scheduler
Synchronization Context Task Scheduler
Task Factories
Task Continuation, Progress and Cancellation
All newer high-level concurrency APIs are all built
on Task.
Task<T> promises to return us a T saying:“not
right now honey, I'm kinda busy, why don't you
come back later? In fact, I will inform you when
I am done or you may cancel me any time you
want”
17. CPU-BOUND
TASK
I Love CPU,
Not You;
Leave Me
Alone
Async Don’ts
Too lightweight I/O (<30ms)
CPU-Intensive Operations
Historic problems with async:
Asynchronous code is difficult
Database backend is a bottleneck
But today ( past few years )
Bottleneck pushed back to app server
18. ASP.NET ASYNC
PATTERNS
Where Were
You?
As If You
Cared? Always
There, Just Bit
Shy
In Asp.Net since very beginning
AsynchronousWeb pages introduced in ASP.NET 2.0
MVC got asynchronous controllers in ASP.NET MVC 2
However, always been awkward to write and
difficult to maintain
Now, the tables have turned
In ASP.NET 4.5, async-await is savior
More and more companies are embracing async
and await on ASP.NET.
25. Old New Description
task.Wait await task Wait/await for a task to complete
task.Result await task Get the result of a completed task
Task.WaitAny await Task.WhenAny Wait/await for one of a collection of tasks to
complete
Task.WaitAll await Task.WhenAll Wait/await for every one of a collection of tasks to
complete
Thread.Sleep await Task.Delay Wait/await for a period of time
Task constructor Task.Run or TaskFactory.StartNew Create a code-based task
26. Problem Solution
Create a task to execute code
Task.Run or TaskFactory.StartNew (not the Task constructor
or Task.Start)
Create a task wrapper for an operation or event TaskFactory.FromAsync or TaskCompletionSource<T>
Support cancellation CancellationTokenSource and CancellationToken
Report progress IProgress<T> and Progress<T>
Handle streams of data TPL Dataflow or Reactive Extensions
Synchronize access to a shared resource SemaphoreSlim
Asynchronously initialize a resource AsyncLazy<T>
Async-ready producer/consumer structures TPL Dataflow or AsyncCollection<T>
27. REENTRANCY
Don’t Exploit Me
For God Sake
Else My Curse
Will Hurt You
Reentering an asynchronous operation before it has
completed
Prevent reentrancy or it can cause unexpected
results
Disable subsequent invokes until its done
Cancel and Restart operation
Run multiple operations and Queue the output
28. FINE TUNING
Want Precision
And Flexibility
To Your Async
App? More APIs
For You
CancellationToken, Task.WhenAll and
Task.WhenAny
Start multiple tasks and await their completion by
monitoring a single task.
Use cases:
Cancel an Async Task or a List of Tasks
Cancel Async Tasks after a Period of Time
Cancel Remaining Async Tasks after One Is Complete
Start Multiple Async Tasks and Process Them As They
Complete
29.
30.
31. To understand why asynchronous requests scale, let’s trace a (simplified) example
of an asynchronous I/O call. Let’s say a request needs to write to a file.The request
thread calls the asynchronous write method.WriteAsync is implemented by the
Base Class Library (BCL), and uses completion ports for its asynchronous I/O. So,
the WriteAsync call is passed down to the OS as an asynchronous file write.The OS
then communicates with the driver stack, passing along the data to write in an I/O
request packet (IRP).
This is where things get interesting: If a device driver can’t handle an IRP
immediately, it must handle it asynchronously. So, the driver tells the disk to start
writing and returns a “pending” response to the OS.The OS passes that “pending”
response to the BCL, and the BCL returns an incomplete task to the request-
handling code.The request-handling code awaits the task, which returns an
incomplete task from that method and so on. Finally, the request-handling code
ends up returning an incomplete task to ASP.NET, and the request thread is freed
to return to the thread pool.
32. Now, consider the current state of the system.There are various I/O structures that
have been allocated (for example, the Task instances and the IRP), and they’re all
in a pending/incomplete state. However, there’s no thread that is blocked waiting
for that write operation to complete. Neither ASP.NET, nor the BCL, nor the OS, nor
the device driver has a thread dedicated to the asynchronous work.
When the disk completes writing the data, it notifies its driver via an interrupt.The
driver informs the OS that the IRP has completed, and the OS notifies the BCL via
the completion port. A thread pool thread responds to that notification by
completing the task that was returned from WriteAsync(); this in turn resumes the
asynchronous request code.
Yes, there were a few threads “borrowed” for very short amounts of time during
this completion-notification phase, but no thread was actually blocked while the
write was in progress.
33. Above example is drastically simplified, but it gets across the primary point: no
thread is required for true asynchronous work. No CPU time is necessary to
actually push the bytes out.
At the device driver level, all non-trivial I/O is asynchronous. Many developers
have a mental model that treats the “natural API” for I/O operations as
synchronous, with the asynchronous API as a layer built on it. However, that’s
completely backward: in fact, the natural API is asynchronous; and it’s the
synchronous APIs that are implemented using asynchronous I/O.
Editor's Notes
AMD: Asynchronous Method Dispatch
Trendy subject: Largest paradigm shift - Sequential to asynchronous programming
If a developer needs to achieve better scalability, they can use any async APIs exposed, and they don’t have to pay additional overhead for invoking a faux async API. If a developer needs to achieve responsiveness or parallelism with synchronous APIs, they can simply wrap the invocation with a method like Task.Run
Scalability benefits is achieved by modifying the actual implementation, whereas offloading can be achieved by just wrapping sync implementations
TAP: System.Threading.Tasks (Task and Task<TResult>)
-In most cases the thread pool will perform better with its own algorithm for allocating threads.
-every process has at least one thread in it
Thread Kernel Object: The OS allocates these data structure to each of the thread created which contains a set of CPU registers. This also contains threads context and consumes space of 700 bytes in X86 and 1240 bytes in X64 bit machines.
Thread Environment Block (TEB): This is the block of memory allocated and initialized in user mode . This consumes 4kb in X84 and 8kb in X64. It helps in storing threads local storage data as well as data structures used in GDI and OpenGL graphics.
User Mode Stack: The user mode stack is used for local variables and arguments passed to methods. It also contains the next statement to be executes when thread returns from method. By default Windows allocates 1MB of memory.
Kernel Mode Stack: It is used when application code passes arguments to Kernel mode function in the OS. This is used mainly for security reasons i.e. when Windows copies any data passed from User mode stack to Kernel mode. Post this process Windows validates the data and operates on them. It consumes 12kb in X86 and 24kb in X64 bit machines.
Worker threads: Async operations as accessing file system, database, services etc.
I/O completion port (IOCP) threads: Used to notify when async operations are completed
Starvation avoidance, the .Net thread pool continues to add worker threads if there is no visible progress on the queued items. In the latter case, the .Net thread pool tries to maximize the throughput using as few threads as possible.
Hence, if your system has four cores, you would have four worker threads and four IOCP threads by default.
Worker threads: Async operations as accessing file system, database, services etc.
I/O completion port (IOCP) threads: Used to notify when async operations are completed
Strategies: Starvation avoidance, the .Net thread pool continues to add worker threads if there is no visible progress on the queued items. In the latter case, the .Net thread pool tries to maximize the throughput using as few threads as possible.
Hence, if your system has four cores, you would have four worker threads and four IOCP threads by default.
W3WP process which ultimately responds to the request.
App pool: application pool is a way to create compartments in a web server through process boundaries, and route sets of URLs to each of these compartments.
This is all well and good—until your ASP.NET server gets more requests than it has threads to handle. At this point, the extra requests have to wait for a thread to be available before they can run.
Those threads are just blocked waiting for an external call to return. They’re not in a running state and are not given any CPU time. Those threads are just being wasted while there’s a request in need. This is the situation addressed by asynchronous requests.
The third request is already in the system. Its timer is going, and it’s in danger of an HTTP Error 503 (Service unavailable).
All green
Asynchronous code does not replace the thread pool, rather makes optimum use of it
Thread pool has a limited injection rate, a thread per 0.5 second
What About the Thread Doing the Asynchronous Work? Async code frees up the request thread, but only at the expense of another thread elsewhere in the system, right?
All newer high-level concurrency APIs, including the Parallel.For*() methods, PLINQ, C# 5 await, and modern async methods in the BCL, are all built on Task.
Not A Silver Bullet. Well, Nothing in this world Is Except silver bullet itself.
Async and await is all about I/O: Excel at reading and writing files, database records, and REST APIs
Two valid arguments
(thus more developer time compared to just purchasing larger servers); and second
Scaling the app server makes little sense if the database back end is the bottleneck
Modern back ends such as Microsoft Azure SQL Database, NoSQL etc. scale much further, pushing the bottleneck back to the Web server
Many companies decided it was easier all around to just develop the code synchronously and pay for larger server farms or more expensive hosting
in ASP.NET 4.5, asynchronous code using async and await is almost as easy as writing synchronous code.
As large systems move into cloud hosting and demand more scaling, more and more companies are embracing async and await on ASP.NET.
TPL – Effort on Natural Parallelism
Make Your Code Span Across CPU Cores
WhenAny returns a task that completes when any task in a collection is complete.
WhenAll returns a task that completes when all tasks in a collection are complete.
Use cases:
Cancel an Async Task or a List of Tasks (C#).
Cancel Async Tasks after a Period of Time (C#)
Cancel Remaining Async Tasks after One Is Complete (C#)
Start Multiple Async Tasks and Process Them As They Complete (C#)
I/O request packets (IRPs) are kernel mode structures that are used by Windows Driver Model (WDM) and Windows NT device drivers to communicate with each other and with the operating system. They are data structures that describe I/O requests, and can be equally well thought of as "I/O request descriptors" or similar.