Highly concurrent yet natural programming

Highly concurrent yet natural
programming
mefyl
quentin.hocquet@infinit.io
Version 1.2

Infinit & me
Me
• Quentin "mefyl" Hocquet
• Epita CSI (LRDE) 2008.
• Ex Gostai
• Into language theory
• Joined Infinit early two years ago.

Infinit & me
Me
• Quentin "mefyl" Hocquet
• Epita CSI (LRDE) 2008.
• Ex Gostai
• Into language theory
• Joined Infinit early two years ago.
Infinit
• Founded my Julien "mycure" Quintard, Epita SRS 2007
• Based on his thesis at Cambridge
• Decentralized filesystem in byzantine environment
• Frontend: file transfer application based on the technology.
• Strong technical culture

Concurrent and parallel
programming

Know the difference
Parallel programming
Aims at running two tasks simultaneously. It is a matter of performances.
Concurrent programming
Aims at running two tasks without inter-blocking. It is a matter of behavior.

Task 1 Task 2
Know the difference

Task 1 Task 2
Know the difference
Sequential

Task 1 Task 2
Know the difference
Parallel

Task 1 Task 2
Know the difference
Concurrent

Sequential Concurrent
Know the difference
Parallel

Know the difference
Parallel
Sequential Concurrent Parallel
CPU usage N N N
Execution time Long Short Shorter

Know the difference
Parallel
Sequential Concurrent Parallel
CPU usage N N N
Execution time Long Short Shorter
Need to run in parallel No No Yes

TV
Commercials
TV
Peeling
Some real life examples
You are the CPU. You want to:
• Watch a film on TV.
• Peel potatoes.

Sequential
TV
Commercials
TV
Peeling
Concurrent
TV
Peeling
TV
Peeling
Parallel
TV Peeling
Commercials
TV

Load
Unload
Load
Unload
You are the CPU. You want to:
• Do the laundry.
• Do the dishes.

Sequential
Load
Unload
Load
Unload
Concurrent
Load
Load
Unload
Unload
Parallel
Load Load
Unload Unload

Some programming examples
Video encoding: encode a raw 2GB raw file to mp4.
• CPU bound.
• File chunks can be encoded separately and then merged later.

Parallel
Encode
first half
Encode
second half
Sequential
Concurrent
Encode first
half
Encode
second half
Video encoding: encode a raw 2GB raw file to mp4.
• CPU bound.
• File chunks can be encoded separately and then merged later.
Parallelism is a plus, concurrency doesn't apply.

An IRC server: handle up to 50k IRC users chatting.
• IO bound.
• A huge number of clients that must be handled concurrently and mostly
waiting.

Concurrent Parallel
An IRC server: handle up to 50k IRC users chatting.
• IO bound.
• A huge number of clients that must be handled concurrently and mostly
waiting.
Concurrency is needed, parallelism is superfluous.

Know the difference
Parallelism
• Is never needed for correctness.
• Is about performances, not correct behavior.
• Is about exploiting multi-core and multi-CPU architectures.
• Can be needed for correctness.
• Is about correct behavior, sometimes about performances too.
• Is about multiple threads being responsive in concurrent.

Know the difference
Parallelism
A good video encoding app:
• Encodes 4 times faster on a 4-core CPU. That's parallelism.

Know the difference
Parallelism
A good video encoding app:
• Encodes 4 times faster on a 4-core CPU. That's parallelism.
• Has a responsive GUI while encoding. That's concurrency.

Who's best ?
If you are parallel, you are concurrent. So why bother ?

Who's best ?
• Being parallel is much, much more difficult. That's time, money and
programmer misery.

Who's best ?
• Being parallel is much, much more difficult. That's time, money and
programmer misery.
• You can't be efficiently parallel past your hardware limit. Those are system
calls, captain.

Threads, callbacks
So, how do you write an echo server ?

The sequential echo server
TCPServer server;
server.listen(4242);
while (true)
{
TCPSocket client = server.accept();
}

TCPServer server;
while (true)
{
while (true)
{
std::string line = client.read_until("n");
client.send(line);
}
}

TCPServer server;
while (true)
{
try
{
while (true)
{
client.send(line);
}
}
catch (ConnectionClosed const&)
{}
}

TCPServer server;
while (true)
{
serve_client(client);
}

TCPServer server;
while (true)
{
}
• Dead simple: you got it instantly. It's natural.
• But wrong: we handle only one client at a time.
• We need ...

TCPServer server;
while (true)
{
}
• Dead simple: you got it instantly. It's natural.
• But wrong: we handle only one client at a time.
• We need ... concurrency !

The parallel echo server
TCPServer server;
while (true)
{
}

TCPServer server;
std::vector<std::thread> threads;
while (true)
{
std::thread client_thread(
[&]
{
});
client_thread.run();
vectors.push_back(std::move(client_thread));
}

TCPServer server;
std::vector<std::thread> threads;
while (true)
{
[&]
{
});
client_thread.run();
vectors.push_back(std::move(client_thread));
}
• Almost as simple and still natural,
• To add the concurrency property, we just added a concurrency construct
to the existing.

But parallelism is too much
• Not scalable: you can't run 50k threads.

• Induces unwanted complexity: race conditions.

int line_count = 0;
while (true)
{
while (true)
{
client.send(line);
++line_count;
}
}

int line_count = 0;
while (true)
{
[&]
{
while (true)
{
client.send(line);
++line_count;
}
});
}

We need concurrency without threads
We need to accept, read and write to socket without threads so without
blocking.

blocking.
• Use select to monitor all sockets at once.
• Register actions to be done when something is ready.
• Wake up only when something needs to be performed.

blocking.
• Use select to monitor all sockets at once.
• Register actions to be done when something is ready.
• Wake up only when something needs to be performed.
This is abstracted with the reactor design pattern:
• libevent
• Boost ASIO
• Python Twisted
• ...

The callback-based echo server
Reactor reactor;
TCPServer server(reactor);
server.accept(&handle_connection);
reactor.run();

Reactor reactor;
reactor.run();
void
handle_connection(TCPSocket& client)
{
client.read_until("n", &handle_read);
}

Reactor reactor;
reactor.run();
void
handle_connection(TCPSocket& client);
void
handle_read(TCPSocket& c, std::string const& l, Error e)
{
if (!e)
c.send(l, &handle_sent);
}

Reactor reactor;
reactor.run();
void
void
handle_read(TCPSocket& c, std::string const& l, Error e);
void
handle_sent(TCPSocket& client, Error error)
{
if (!e)
client.read_until("n", &handle_read);
}

How do we feel now ?
• This one scales to thousands of client.

• Yet to add the concurrency property, we had to completely change the way
we think.

• Yet to add the concurrency property, we had to completely change the way
we think.
• A bit more verbose and complex, but nothing too bad ... right ?

Counting lines with threads
try
{
while (true)
{
client.send(line);
}
}
{
}

Counting lines with threads
int lines_count = 0;
try
{
while (true)
{
++lines_count;
client.send(line);
}
}
{
std::cerr << "Client sent " << lines_count << "linesn";
}

Counting lines with callbacks
void
handle_connection(TCPSocket& client)
{
int* count = new int(0);
client.read_until(
"n", std::bind(&handle_read, count));
}

void
void
handle_read(TCPSocket& c, std::string const& l,
Error e, int* count)
{
if (e)
std::cerr << *count << std::endl;
else
c.send(l, std::bind(&handle_sent, count));
}

void
void
handle_read(TCPSocket& c, std::string const& l,
Error e, int* count);
void
handle_sent(TCPSocket& client, Error error, int* count)
{
if (e)
std::cerr << *count << std::endl;
else
client.read_until(
"n", std::bind(&handle_read, count));
}

Callback-based programming considered harmful
• Code is structured with callbacks.

• Asynchronous operation break the flow arbitrarily.

• You lose all syntactic scoping expression (local variables, closure,
exceptions, ...).

• You lose all syntactic scoping expression (local variables, closure,
exceptions, ...).
• This is not natural. Damn, this is pretty much as bad as GOTO.

Are we screwed ?
Threads
• Respect your beloved semantic and expressiveness.
• Don't scale and introduce race conditions.

Are we screwed ?
Threads
Callbacks
• Scale.
• Ruins your semantic. Painful to write, close to impossible to maintain.

Are we screwed ?
Threads
Callbacks
• Scale.
I lied when I said: we need concurrency without threads.

Are we screwed ?
Threads
Callbacks
• Scale.
I lied when I said: we need concurrency without threads.
We need concurrency without system threads.

Coroutines
Also known as:
• green threads
• userland threads
• fibers
• contexts
• ...

Coroutines
• Separate execution contexts like system threads.
• Userland: no need to ask the kernel.
• Non-parallel.
• Cooperative instead of preemptive: they yield to each other.

Coroutines
• Separate execution contexts like system threads.
• Userland: no need to ask the kernel.
• Non-parallel.
• Cooperative instead of preemptive: they yield to each other.
By building on top of that, we have:
• Scalability: no system thread involved.
• No arbitrary race-conditions: no parallelism.
• A stack, a context: the code is natural.

Coroutines-based scheduler
• Make a scheduler that holds coroutines .
• Embed a reactor in there.
• Write a neat Socket class.

Coroutines-based scheduler
• Make a scheduler that holds coroutines .
• Embed a reactor in there.
• Write a neat Socket class. When read, it:
◦ Unschedules itself.
◦ Asks the reactor to read
◦ Pass a callback to reschedule itself
◦ Yield control back.

Coroutines-based echo server
TCPServer server; server.listen(4242);
std::vector<Thread> threads;
int lines_count = 0;
while (true)
{
Thread t([client = std::move(client)] {
try
{
while (true)
{
++lines_count;
client.send(client.read_until("n"));
}
}
catch (ConnectionClosed const&) {}
});
threads.push_back(std::move(t));
}

What we built at Infinit: the reactor.

What we built at Infinit: the reactor.
• Coroutine scheduler: simple round robin
• Sleeping, waiting
• Timers
• Synchronization
• Mutexes, semaphores
• TCP networking
• SSL
• UPnP
• HTTP client (Curl based)

Coroutine scheduling
reactor::Scheduler sched;
reactor::Thread t1(sched,
[&]
{
print("Hello 1");
reactor::yield();
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("Bye 2");
});
);
sched.run();

Coroutine scheduling
reactor::Scheduler sched;
[&]
{
print("Hello 1");
reactor::yield();
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("Bye 2");
});
);
sched.run();
Hello 1
Hello 2
Bye 1
Bye 2

Sleeping and waiting
[&]
{
print("Hello 1");
reactor::sleep(500_ms);
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("World 2");
reactor::yield();
print("Bye 2");
});
);

[&]
{
print("Hello 1");
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("World 2");
reactor::yield();
print("Bye 2");
});
);
Hello 1
Hello 2
World 2
Bye 2

[&]
{
print("Hello 1");
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("World 2");
reactor::yield();
print("Bye 2");
});
);
Hello 1
Hello 2
World 2
Bye 2
Bye 1

[&]
{
print("Hello 1");
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("World 2");
reactor::wait(t1); // Wait
print("Bye 2");
});
);

[&]
{
print("Hello 1");
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("World 2");
print("Bye 2");
});
);
Hello 1
Hello 2
World 2

[&]
{
print("Hello 1");
print("Bye 1");
});
[&]
{
print("Hello 2");
reactor::yield();
print("World 2");
print("Bye 2");
});
);
Hello 1
Hello 2
World 2
Bye 1
Bye 2

Synchronization: signals
reactor::Signal task_available;
std::vector<Task> tasks;
reactor::Thread handler([&] {
while (true)
{
if (!tasks.empty())
{
std::vector mytasks = std::move(tasks);
for (auto& task: tasks)
; // Handle task
}
else
reactor::wait(task_available);
}
});

while (true)
{
if (!tasks.empty())
{
; // Handle task
}
else
reactor::wait(task_available);
}
});
tasks.push_back(...);
task_available.signal();

while (true)
{
if (!tasks.empty()) // 1
{
; // Handle task
}
else
reactor::wait(task_available); // 4
}
});
tasks.push_back(...); // 2
task_available.signal(); // 3

Synchronization: channels
reactor::Channel<Task> tasks;
while (true)
{
Task t = tasks.get();
// Handle task
}
});
tasks.put(...);

Mutexes
But you said no race conditions! You lied again!

Mutexes
reactor::Thread t([&] {
while (true)
{
for (auto& socket: sockets)
socket.send("YO");
}
});
{
socket.push_back(...);
}

Mutexes
reactor::Mutex mutex;
while (true)
{
reactor::wait(mutex);
socket.send("YO");
mutex.unlock();
}
});
{
reactor::wait(mutex);
mutex.unlock();
}

Mutexes
reactor::Mutex mutex;
while (true)
{
reactor::Lock lock(mutex);
socket.send("YO");
}
});
{
reactor::Lock lock(mutex);
}

Networking: TCP
We saw a good deal of TCP networking:
try
{
reactor::TCPSocket socket("battle.net", 4242, 10_sec);
// ...
}
catch (reactor::network::ResolutionFailure const&)
{
// ...
}
catch (reactor::network::Timeout const&)
{
// ...
}

Networking: TCP
We saw a good deal of TCP networking:
void
serve(TCPSocket& client)
{
try
{
std::string auth = server.read_until("n", 10_sec);
if (!check_auth(auth))
// Impossible with callbacks
throw InvalidCredentials();
while (true) { ... }
}
catch (reactor::network::Timeout const&)
{}
}

Networking: SSL
Transparent client handshaking:
reactor::network::SSLSocket socket("localhost", 4242);
socket.write(...);

Networking: SSL
Transparent server handshaking:
reactor::network::SSLServer server(certificate, key);
while (true)
{
auto socket = server.accept();
reactor::Thread([&] { ... });
}

Networking: SSL
SSLSocket SSLServer::accept()
{
auto socket = this->_tcp_server.accept();
// SSL handshake
return socket
}

Networking: SSL
reactor::Channel<SSLSocket> _sockets;
void SSLServer::_handshake_thread()
{
while (true)
{
// SSL handshake
this->_sockets.put(socket);
}
}
SSLSocket SSLServer::accept()
{
return this->_accepted.get;
}

Networking: SSL
void SSLServer::_handshake_thread()
{
while (true)
{
reactor::Thread t(
[&]
{
// SSL handshake
this->_sockets.put(socket);
});
}
}

HTTP
std::string google = reactor::http::get("google.com");

HTTP
reactor::http::Request r("kissmetrics.com/api",
reactor::http::Method::PUT,
"application/json",
5_sec);
r.write("{ event: "login"}");
reactor::wait(r);

HTTP
reactor::http::Request r("kissmetrics.com/api",
reactor::http::Method::PUT,
"application/json",
5_sec);
r.write("{ event: "login"}");
reactor::wait(r);
• Chunking
• Cookies
• Custom headers
• Upload/download progress
• ... pretty much anything Curl supports (i.e., everything)

HTTP streaming
std::string content = reactor::http::get(
"my-api.infinit.io/transactions");
auto json = json::parse(content);

HTTP streaming
reactor::http::Request r(
"my-api.production.infinit.io/transactions");
assert(r.status() == reactor::http::Status::OK);
// JSON is parsed on the fly;
auto json = json::parse(r);

HTTP streaming
assert(r.status() == reactor::http::Status::OK);
// JSON is parsed on the fly;
auto json = json::parse(r);
"youtube.com/upload", http::reactor::Method::PUT);
std::ifstream input("~/A new hope - BrRIP.mp4");
std::copy(input, r);

Better concurrency: futures, ...
std::string transaction_id = reactor::http::put(
// Ask the user files to share.
reactor::http::post("my-api.infinit.io/transaction/", file_list);
std::string s3_token = reactor::http::get(
"s3.aws.amazon.com/get_token?key=...");
// Upload files to S3

std::string transaction_id = reactor::http::put(
reactor::http::post("my-api.infinit.io/transaction/", file_list);
std::string s3_token = reactor::http::get(
reactor::http::Request transaction(
reactor::http::Request s3(
auto transaction_id = transaction.content();
reactor::http::Request list(
"my-api.infinit.io/transaction/", file_list);
auto s3_token = transaction.content();

Version 1
Wait meta
Ask files
Wait meta
Wait AWS
Version 2
Ask files
Version 2
Ask files

How does it perform for us ?
• Notification server does perform:
◦ 10k clients per instance
◦ 0.01 load average
◦ 1G resident memory
◦ Cheap monocore 2.5 Ghz (EC2)

How does it perform for us ?
• Notification server does perform:
◦ 10k clients per instance
◦ 0.01 load average
◦ 1G resident memory
◦ Cheap monocore 2.5 Ghz (EC2)
• Life is so much better:
◦ Code is easy and pleasant to write and read
◦ Everything is maintainable
◦ Send metrics on login without slowdown? No biggie.
◦ Try connecting to several interfaces and keep the first to respond? No
biggie.

Highly concurrent yet natural programming

More Related Content

What's hot

Viewers also liked

Similar to Highly concurrent yet natural programming

Recently uploaded

Highly concurrent yet natural programming