Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Infinit filesystem, Reactor reloaded

619 views

Published on

Infinit's C++ coroutine-based asynchronous framework has been the fundamental brick on top of which the peer-to-peer POSIX-compliant file system has been built.

Published in: Software
  • Be the first to comment

Infinit filesystem, Reactor reloaded

  1. 1. Infinit filesystem Reactor reloaded mefyl quentin.hocquet@infinit.io Version 1.2-5-g4a755e6
  2. 2. Infinit filesystem Distributed filesystem in byzantine environment: aggregate multiple computers into one storage.
  3. 3. Infinit filesystem Distributed filesystem in byzantine environment: aggregate multiple computers into one storage. • Filesystem in the POSIX sense of the term. ◦ Works with any client, unmodified. ◦ Fine grained semantic (e.g: video streaming). ◦ Well featured (e.g: file locking).
  4. 4. Infinit filesystem Distributed filesystem in byzantine environment: aggregate multiple computers into one storage. • Filesystem in the POSIX sense of the term. ◦ Works with any client, unmodified. ◦ Fine grained semantic (e.g: video streaming). ◦ Well featured (e.g: file locking). • Distributed: no computer as any specific authority or role. ◦ Availability: no SPOF, network failure resilient. ◦ No admin. As in, no janitor and no tyran. ◦ Scalability flexibility.
  5. 5. Infinit filesystem Distributed filesystem in byzantine environment: aggregate multiple computers into one storage. • Filesystem in the POSIX sense of the term. ◦ Works with any client, unmodified. ◦ Fine grained semantic (e.g: video streaming). ◦ Well featured (e.g: file locking). • Distributed: no computer as any specific authority or role. ◦ Availability: no SPOF, network failure resilient. ◦ No admin. As in, no janitor and no tyran. ◦ Scalability flexibility. • Byzantine: you do not need to trust other computers in any way. ◦ No admins. As in, no omniscient god. ◦ Support untrusted networks, both faulty and malicious peers.
  6. 6. Infinit architecture
  7. 7. Coroutines in a nutshell
  8. 8. Coroutines in a nutshell Intelligible Race conditions Scale Multi core Stack memory System threads OK KO KO OK KO
  9. 9. Coroutines in a nutshell Intelligible Race conditions Scale Multi core Stack memory System threads OK KO KO OK KO Event-based KO OK OK KO OK
  10. 10. Coroutines in a nutshell Intelligible Race conditions Scale Multi core Stack memory System threads OK KO KO OK KO Event-based KO OK OK KO OK System Coroutines OK OK OK KO KO
  11. 11. Coroutines in a nutshell Intelligible Race conditions Scale Multi core Stack memory System threads OK KO KO OK KO Event-based KO OK OK KO OK System Coroutines OK OK OK KO KO Stackless Coroutines Meh OK OK KO OK
  12. 12. Coroutines in a nutshell Intelligible Race conditions Scale Multi core Stack memory System threads OK KO KO OK KO Event-based KO OK OK OK OK System Coroutines OK OK OK OK KO Stackless Coroutines Meh OK OK OK OK
  13. 13. Reactor in a nutshell Reactor is a C++ library providing coroutines support, enabling simple and safe imperative style concurrency.
  14. 14. Reactor in a nutshell Reactor is a C++ library providing coroutines support, enabling simple and safe imperative style concurrency. while (true) { auto socket = tcp_server.accept(); new Thread([socket] { try { while (true) socket->write(socket->read_until("n")); } catch (reactor::network::Error const&) {} }); }
  15. 15. Sugar for common pattern Coroutines are mostly used in three patterns:
  16. 16. Sugar for common pattern Coroutines are mostly used in three patterns: • The entirely autonomous detached thread. • The background thread tied to an object. • The parallel flow threads tied to the stack.
  17. 17. Basic API for threads Three core calls: • Create a thread that will run callable concurrently: reactor::Thread("name", callable)
  18. 18. Basic API for threads Three core calls: • Create a thread that will run callable concurrently: reactor::Thread("name", callable) • Wait until the a thread finishes: reactor::wait(thread)
  19. 19. Basic API for threads Three core calls: • Create a thread that will run callable concurrently: reactor::Thread("name", callable) • Wait until the a thread finishes: reactor::wait(thread) • Terminate a thread: thread->terminate_now()
  20. 20. Basic API for threads Three core calls: • Create a thread that will run callable concurrently: reactor::Thread("name", callable) • Wait until the a thread finishes: reactor::wait(thread) • Terminate a thread: thread->terminate_now() Nota bene: • Don't destroy an unfinished thread. Wait for it or terminate it.
  21. 21. Basic API for threads Three core calls: • Create a thread that will run callable concurrently: reactor::Thread("name", callable) • Wait until the a thread finishes: reactor::wait(thread) • Terminate a thread: thread->terminate_now() Nota bene: • Don't destroy an unfinished thread. Wait for it or terminate it. • Exceptions escaping a thread terminate the whole scheduler.
  22. 22. Basic API for reactor reactor::Scheduler sched; reactor::Thread main( sched, "main", [] { }); // Will execute until all threads are done. sched.run();
  23. 23. Basic API for reactor reactor::Scheduler sched; reactor::Thread main( sched, "main", [] { reactor::Thread t("t", [] { print("world"); }); print("hello"); }); // Will execute until all threads are done. sched.run();
  24. 24. Basic API for reactor reactor::Scheduler sched; reactor::Thread main( sched, "main", [] { reactor::Thread t("t", [] { print("world"); }); print("hello"); reactor::wait(t); }); // Will execute until all threads are done. sched.run();
  25. 25. The detached thread The detached thread is a global background operation whose lifetime is tied to the program only.
  26. 26. The detached thread The detached thread is a global background operation whose lifetime is tied to the program only. E.g. uploading crash reports on startup. for (auto p: list_directory(reports_dir)) reactor::http::put("infinit.sh/reports", ifstream(p));
  27. 27. The detached thread The detached thread is a global background operation whose lifetime is tied to the program only. E.g. uploading crash reports on startup. new reactor::Thread( [] { for (auto p: list_directory(reports_dir)) reactor::http::put("infinit.sh/reports", ifstream(p)); });
  28. 28. The detached thread The detached thread is a global background operation whose lifetime is tied to the program only. E.g. uploading crash reports on startup. new reactor::Thread( [] { for (auto p: list_directory(reports_dir)) reactor::http::put("infinit.sh/reports", ifstream(p)); }, reactor::Thread::auto_dispose = true);
  29. 29. The background thread tied to an object The background thread performs a concurrent operation for an object and is tied to its lifetime
  30. 30. The background thread tied to an object The background thread performs a concurrent operation for an object and is tied to its lifetime class Async { reactor::Channel<Block> _blocks; void store(Block b) { this->_blocks->put(b); } void _flush() { while (true) this->_store(this->_blocks.get()); } };
  31. 31. The background thread tied to an object The background thread performs a concurrent operation for an object and is tied to its lifetime class Async { reactor::Channel<Block> _blocks; void store(Block b); void _flush(); std::unique_ptr<reactor::Thread> _flush_thread; Async() : _flush_thread(new reactor::Thread( [this] { this->_flush(); })) {} };
  32. 32. The background thread tied to an object The background thread performs a concurrent operation for an object and is tied to its lifetime class Async { reactor::Channel<Block> _blocks; void store(Block b); void _flush(); std::unique_ptr<reactor::Thread> _flush_thread; Async(); ~Async() { this->_flush_thread->terminate_now(); } };
  33. 33. The background thread tied to an object The background thread performs a concurrent operation for an object and is tied to its lifetime class Async { reactor::Channel<Block> _blocks; void store(Block b); void _flush(); Thread::unique_ptr<reactor::Thread> _flush_thread; Async(); };
  34. 34. The background thread tied to an object The background thread performs a concurrent operation for an object and is tied to its lifetime struct Terminator : public std::default_delete<reactor::Thread> { void operator ()(reactor::Thread* t) { bool disposed = t->_auto_dispose; if (t) t->terminate_now(); if (!disposed) std::default_delete<reactor::Thread>:: operator()(t); } }; typedef std::unique_ptr<reactor::Thread, Terminator> unique_ptr;
  35. 35. The parallel flow threads Parallel flow threads are used to make the local flow concurrent, like parallel control statements. auto data = reactor::http::get("...");; reactor::http::put("http://infinit.sh/...", data); reactor::network::TCPSocket s("hostname"); s.write(data); print("sent!"); GET HTTP PUT TCP PUT SENT
  36. 36. The parallel flow threads auto data = reactor::http::get("..."); reactor::Thread http_put( [&] { reactor::http::put("http://infinit.sh/...", data); }); reactor::Thread tcp_put( [&] { reactor::network::TCPSocket s("hostname"); s.write(data); }); reactor::wait({http_put, tcp_put}); print("sent!"); GET SENT HTTP PUT TCP PUT
  37. 37. The parallel flow threads auto data = reactor::http::get("..."); std::exception_ptr exn; reactor::Thread http_put([&] { try { reactor::http::put("http://...", data); } catch (...) { exn = std::current_exception(); } }); reactor::Thread tcp_put([&] { try { reactor::network::TCPSocket s("hostname"); s.write(data); } catch (...) { exn = std::current_exception(); } }); reactor::wait({http_put, tcp_put}); if (exn) std::rethrow_exception(exn); print("sent!");
  38. 38. The parallel flow threads auto data = reactor::http::get("..."); std::exception_ptr exn; reactor::Thread http_put([&] { try { reactor::http::put("http://...", data); } catch (reactor::Terminate const&) { throw; } catch (...) { exn = std::current_exception(); } }); reactor::Thread tcp_put([&] { try { reactor::network::TCPSocket s("hostname"); s.write(data); } catch (reactor::Terminate const&) { throw; } catch (...) { exn = std::current_exception(); } }); reactor::wait({http_put, tcp_put}); if (exn) std::rethrow_exception(exn); print("sent!");
  39. 39. The parallel flow threads auto data = reactor::http::get("..."); reactor::Scope scope; scope.run([&] { reactor::http::put("http://infinit.sh/...", data); }); scope.run([&] { reactor::network::TCPSocket s("hostname"); s.write(data); }); reactor::wait(scope); print("sent!"); GET SENT HTTP PUT TCP PUT
  40. 40. Futures auto data = reactor::http::get("..."); reactor::Scope scope; scope.run([&] { reactor::http::Request r("http://infinit.sh/..."); r.write(data); }); scope.run([&] { reactor::network::TCPSocket s("hostname"); s.write(data); }); reactor::wait(scope); print("sent!");
  41. 41. Futures elle::Buffer data; reactor::Thread fetcher( [&] { data = reactor::http::get("..."); }); reactor::Scope scope; scope.run([&] { reactor::http::Request r("http://infinit.sh/..."); reactor::wait(fetcher); r.write(data); }); scope.run([&] { reactor::network::TCPSocket s("hostname"); reactor::wait(fetcher); s.write(data); }); reactor::wait(scope); print("sent!");
  42. 42. Futures reactor::Future<elle::Buffer> data( [&] { data = reactor::http::get("..."); }); reactor::Scope scope; scope.run([&] { reactor::http::Request r("http://infinit.sh/..."); r.write(data.get()); }); scope.run([&] { reactor::network::TCPSocket s("hostname"); s.write(data.get()); }); reactor::wait(scope); print("sent!");
  43. 43. Futures Scope + futures SENT GET HTTP PUT TCP PUT Scope GET SENT HTTP PUT TCP PUT Serial GET HTTP PUT TCP PUT SENT
  44. 44. Concurrent iteration The final touch. std::vector<Host> hosts; reactor::Scope scope; for (auto const& host: hosts) scope.run([] (Host h) { h->send(block); }); reactor::wait(scope);
  45. 45. Concurrent iteration The final touch. std::vector<Host> hosts; reactor::Scope scope; for (auto const& host: hosts) scope.run([] (Host h) { h->send(block); }); reactor::wait(scope); Concurrent iteration: std::vector<Host> hosts; reactor::concurrent_for( hosts, [] (Host h) { h->send(block); });
  46. 46. Concurrent iteration The final touch. std::vector<Host> hosts; reactor::Scope scope; for (auto const& host: hosts) scope.run([] (Host h) { h->send(block); }); reactor::wait(scope); Concurrent iteration: std::vector<Host> hosts; reactor::concurrent_for( hosts, [] (Host h) { h->send(block); }); • Also available: reactor::concurrent_break(). • As for a concurrent continue, that's just return.
  47. 47. CPU bound operations In the cases we are CPU bound, how to exploit multicore since coroutines are concurrent but not parallel ?
  48. 48. CPU bound operations In the cases we are CPU bound, how to exploit multicore since coroutines are concurrent but not parallel ? • Idea 1: schedule coroutines in parallel ◦ Pro: absolute best parallelism. ◦ Cons: race conditions.
  49. 49. CPU bound operations In the cases we are CPU bound, how to exploit multicore since coroutines are concurrent but not parallel ? • Idea 1: schedule coroutines in parallel ◦ Pro: absolute best parallelism. ◦ Cons: race conditions. • Idea 2: isolate CPU bound code in "monads" and run it in background. ◦ Pro: localized race conditions. ◦ Cons: only manually parallelized code exploits multiple cores.
  50. 50. reactor::background reactor::background(callable): run callable in a system thread and return its result.
  51. 51. reactor::background reactor::background(callable): run callable in a system thread and return its result. Behind the scene: • A boost::asio_service whose run method is called in as many threads as there are cores. • reactor::background posts the action to Asio and waits for completion.
  52. 52. reactor::background reactor::background(callable): run callable in a system thread and return its result. Behind the scene: • A boost::asio_service whose run method is called in as many threads as there are cores. • reactor::background posts the action to Asio and waits for completion. Block::seal(elle::Buffer data) { this->_data = reactor::background( [data=std::move(data)] { return encrypt(data); }); }
  53. 53. reactor::background No fiasco rules of thumb: be pure. • No side effects. • Take all bindings by value. • Return any result by value.
  54. 54. reactor::background No fiasco rules of thumb: be pure. • No side effects. • Take all bindings by value. • Return any result by value. std::vector<int> ints; int i = 0; // Do not: reactor::background([&] {ints.push_back(i+1);}); reactor::background([&] {ints.push_back(i+2);}); // Do: ints.push_back(reactor::background([i] {return i+1;}); ints.push_back(reactor::background([i] {return i+2;});
  55. 55. Background pools Infinit generates a lot of symmetric keys. Let other syscall go through during generation with reactor::background: Block::Block() : _block_keys(reactor::background( [] { return generate_keys(); })) {}
  56. 56. Background pools Infinit generates a lot of symmetric keys. Let other syscall go through during generation with reactor::background: Block::Block() : _block_keys(reactor::background( [] { return generate_keys(); })) {} Problem: because of usual on-disk filesystems, operations are often sequential. $ time for i in $(seq 128); touch $i; done The whole operation will still be delayed by 128 * generation_time.
  57. 57. Background pools Solution: pregenerate keys in a background pool. reactor::Channel<KeyPair> keys; keys.max_size(64); reactor::Thread keys_generator([&] { reactor::Scope keys; for (int i = 0; i < NCORES; ++i) scope.run([] { while (true) keys.put(reactor::background( [] { return generate_keys(); })); }); reactor::wait(scope); }); Block::Block() : _block_keys(keys.get()) {}
  58. 58. Generators Fetching a block with paxos: • Ask the overlay for the replication_factor owners of the block. • Fetch at least replication_factor / 2 + 1 versions. • Paxos the result
  59. 59. Generators Fetching a block with paxos: DHT::fetch(Address addr) { auto hosts = this->_overlay->lookup(addr, 3); std::vector<Block> blocks; reactor::concurrent_for( hosts, [&] (Host host) { blocks.push_back(host->fetch(addr)); if (blocks.size() >= 2) reactor::concurrent_break(); }); return some_paxos_magic(blocks); }
  60. 60. Generators reactor::Generator<Host> Kelips::lookup(Address addr, factor f) { return reactor::Generator<Host>( [=] (reactor::Generator<Host>::Yield const& yield) { some_udp_socket.write(...); while (f--) yield(kelips_stuff(some_udp_socket.read())); }); }
  61. 61. Generators Behind the scene: template <typename T> class Generator { reactor::Channel<T> _values; Generator(Callable call) : _thread([this] { call([this] (T e) {this->_values.put(e);}); }) {} // Iterator interface calling _values.get() }; • Similar to python generators • Except generation starts right away and is not tied to iteration
  62. 62. Questions?

×