Essentials of Multithreaded System Programming in C++


Published on

Published in: Technology, Education

Essentials of Multithreaded System Programming in C++

  1. 1. Essentials of MultithreadedSystem Programming in C++<br />Shuo Chen<br />2011/02<br /><br /><br />@bnu_chenshuo<br />
  2. 2. Contents<br />Challenges in multithreaded system programming<br />Thread safety of C and C++ libraries<br />RAII and fork()<br />fork() and signal handling in multithreaded programs<br />2011/02<br />Shuo Chen (<br />2<br />
  3. 3. Audience: C++ programmers<br />Familiar with Pthreads and Sockets API<br />Knows thread safety, deadlock, race condition, etc.<br />In a word: read through APUE2e and UNP3e (vol. 1) by W. Richard Stevens et al.<br />All discussions are based on Linux 2.6.x, x >= 28<br />There are new syscalls, egsignalfd, eventfd, and timerfd<br />x86 and x64 platforms<br />2011/02<br />Shuo Chen (<br />3<br />
  4. 4. Multi-threaded system programming<br />Multithreading is inevitable in this multi-core era<br />The difficulties are not learning synchronization primitives (mutexes, condition variables)<br />~10 functions are sufficient to do it right<br />But understanding interactions between existing system calls and library functions<br />Understands how threads affect system design<br />Use it wisely and effectively<br />Avoid common pitfalls and fallacies<br />2011/02<br />Shuo Chen (<br />4<br />
  5. 5. 11 essential Pthreads functions<br />11 out of 110+ pthreads functions<br />2 -> create and join threads<br />4 -> init/destroy, lock/unlock mutexes<br />5 -> init/destroy, wait/signal/broadcast condvars<br />Think twice if you need more<br />Some are okay, eg. once and key, maybe rwlock<br />Some are bad, eg. cancel and kill, semaphores<br />Check muduo/base for encapsulation in C++<br /><br /> click thread<br />2011/02<br />Shuo Chen (<br />5<br />
  6. 6. An asynchronous world<br />Never assume the sequence of events without proper synchronization.<br />Knows happens-before relation, memory visibility, etc.<br />The effect of an interaction between two [thread]s must be independent of the speed at which it is carried out. --- Brinch Hansen 1973<br />2011/02<br />Shuo Chen (<br />6<br />
  7. 7. Standards and practices<br />Although the latest official standards of C and C++ languages (C99 and C++03) do not say a word about process or thread<br />We write multi-process and/or multi-threaded C/C++ programs in real life, as a real-world need<br />We can’t wait it to be standardized, as standards usually fall behind practices for years<br />btw, if there are not real life multi-threads programs , how do people what/how to standardize?<br />We adhere to some de facto standards<br />A lot simpler if we focus on one hardware and one OS<br />2011/02<br />Shuo Chen (<br />7<br />
  8. 8. Thread identifier on Linux<br />Use pid_t as thread id, instead of pthread_t, on Linux<br />pthread_tthid = pthread_self(), thid is opaque (uintptr_t)<br />pid_ttid = ::gettid(), tid is task id, usually a small integer<br />/proc/tid/, /proc/pid/task/tid/, ps, top all work fine<br />How to implement gettid() efficiently? Thread local?<br />gettid(2) is a syscall, but the output should never change<br />getpid(2) caches the result, should gettid() do the same?<br />What if fork(), will it caches the old value in child proc?<br />How about pthread_atfork() to clear it up?<br />Check muduo/base/ for details<br />2011/02<br />Shuo Chen (<br />8<br />
  9. 9. Creation of threads<br />A library should not create its own ‘background’ thread without prior informed consent<br />Makes a program non-forkable<br />Never create thread before main()<br />Avoid creating thread in ctor of static or global object<br />Breaks static objects constructing, eg. protobuf registering<br />The number of threads created should be independentof system load, eg. # of connections, # of requests<br />otherwise non-scalable<br />Reuse threads, by assign multiple roles to it<br />Doing IO and timer with muduoEventLoop class<br />For simple task, do it within IO callbacks in IO threads<br />2011/02<br />Shuo Chen (<br />9<br />
  10. 10. Three ways of termination<br /><br />Natural death – return from thread function, good<br />Suicide – call pthread_exit()<br />Mudered – killed by pthread_cancel()<br />Rule: let it die, never suicide or murder a thread<br />Why? inherently deadlock-prone: no chance to unlock<br />Design your program so that a thread can be waken up and safely exits<br />For reference<br />Java Thread.{stop, suspend and destroy} are deprecated<br />Boost Threads doesn’t provide thread::cancel()<br />2011/02<br />Shuo Chen (<br />10<br />
  11. 11. pthread_cancel() and C++<br />In C, we have concept of ‘cancellation point’<br />In C++, pthread_cancel() throws an exception in that thread, helps unwinding objects on stack<br />The exception must reach the outmost function, otherwise core dump:<br />FATAL: exception not rethrown<br />Aborted (core dumped)<br />Always rethrow in catch(…) cause<br />Ulrich Drepper “Cancellation and C++ Exceptions”<br />Better: never cancel or kill a thread<br />2011/02<br />Shuo Chen (<br />11<br />
  12. 12. exit() is not thread safe in C++<br />exit() destructs static or global objects, (_exit() doesn’t)<br />The destructor may try to hold a lock<br />The caller function may have held the same lock already<br />End up in a dead lock<br />Check following code for an example of dead lock<br /><br />How to quit a multi-threaded program safely?<br />An irregular but simple solution: make a process killable, eg.<br />p.29<br />It’s not fault of exit(), but static or global objects<br />Try to avoid static or global objects in C++, except for PODs<br />2011/02<br />Shuo Chen (<br />12<br />
  13. 13. Thread local __thread in g++<br />Thread safe by natural, unless escaped to other thread<br />More efficient implementation, than pthread_key_t<br />See “ELF Handling For Thread-Local Storage”<br />In C++, must be initialized with constant-expression<br />No __thread string t_obj("Chen Shuo");<br />No __thread string* t_obj = new string;<br />Only __thread string* t_obj = NULL;<br />More rules: <br /><br />Use pthread_key_t if you want auto destruction<br />2011/02<br />Shuo Chen (<br />13<br />
  14. 14. Use non-recursive mutex only<br />A basic assumption of holding a mutex<br />Once I lock it, I can modify the guarded object safely<br />Which is not true for recursive mutex, eg.<br /><br />Recursive mutexes by David Butenhof<br /> <br />Recursive locks - a blessing or a curse?  <br /><br />2011/02<br />Shuo Chen (<br />14<br />
  15. 15. Impacts of introducing threads<br />Threading is a late patch to OS kernel<br />Unix kernel and API formed in early 1970s<br />First implementation of threads emerged in early 1990s<br />Breaks lots of assumptions made during the 20 years<br />Library functions with side effects must be revisited<br />malloc/free, fread/fseekcan be made thread-safe with locks<br />Functions that return or use static allocated space are not thread safe but may have thread-safe variants<br />asctime_r, ctime_r, gmtime_r, rand_r, stderror_r, strtok_r<br />errno is not an ‘extern int’, but a per-thread value<br />extern int *__errno_location(void);<br />#define errno (*__errno_location())<br />2011/02<br />Shuo Chen (<br />15<br />
  16. 16. Thread safety of C library<br />Individual system calls must be thread safe<br />Be caution of interfering of same file descriptor from multiple threads<br />Most of glibc library functions are thread safe nowadays<br />Counterintuitively, Posix standards lists functions thatare not required to be thread safe, it's a black list.<br /><br />2.9.1 Thread-Safety :<br />All functions defined by this volume of POSIX.1-2008 shall be thread-safe, except that the following functions need not be thread-safe.<br />Notably, getenv/putenv/setenv/system() are not safe<br />2011/02<br />Shuo Chen (<br />16<br />
  17. 17. FILE* functions are thread safe<br />Read ‘man flockfile’, but they are not composable, eg.<br />fseek(), followed by fread()<br />The file position may change during the course by a different thread<br />Wrap with flockfile(FILE*) and funlockfile(FILE*) <br />Same applies to lseek(2) and read(2), but how to lock?<br />Use pread(2) instead, which doesn’t change the file offset<br />In general, a function that calls two thread-safe functions is not guaranteed to be thread-safe<br />Just like exception-safety, thread-safety is not composable<br />2011/02<br />Shuo Chen (<br />17<br />
  18. 18. Thread safety is not composable<br />A solution works in single-threaded program may not apply to multi-threaded program.<br />Any solution calls two or more thread safe function are not necessarily correct in multi-threaded program<br />What’s the time in London now? Program runs in New York<br />string oldTz = getenv("TZ"); // save TZ<br />putenv("TZ=Europe/London"); tzset(); // set TZ to London<br />struct tm localTimeInLN = *localtime(time(NULL));<br />setenv("TZ", oldTz.c_str(), 1); tzset(); // restore old TZ<br />This code impacts localtime() in other threads<br />Thread safe functions are not composable unless you carefully design the interface and interactions<br />2011/02<br />Shuo Chen (<br />18<br />
  19. 19. Thread safety of C++ std library<br />Although not required by the standard, the de facto says<br />Unshared objects are independent: Two threads can freely use different objects without any special action on the caller's part. We call it "same level as built-in types."<br />This applies to STL containers like map, vector, string<br />Pure functions are safe, eg. Most of STL algorithms.<br />The global cin/cout objects are shared by threads, and are not thread safe. Moreover, they can't be made safe<br />cout << a << b;  cout.operator<<(a).operator<<(b);<br />Two function calls can be interrupted by another thread<br />Use printf(3) instead, it's thread safe and atomic.<br />Allocators must be thread safe, as they are shared<br />2011/02<br />Shuo Chen (<br />19<br />
  20. 20. Thread-Safe vs. Thread-Efficient<br />printf(3) and malloc(3) are thread safe, but not necessarily efficient enough, esp. on multi-cores<br />printf(3) locks FILE* stdout, synchronizes threads<br />not good for multi-threaded logging, we need a better lib<br />your default malloc(3) may not optimized for multi-threads and multi-cores<br />it may lock global heap for each allocation<br />try tcmalloc, Google's thread-cache malloc<br />see Intel. Is your memory management multi-core ready?<br /><br />2011/02<br />Shuo Chen (<br />20<br />
  21. 21. Operate one fd in one thread<br />Although system calls of file descriptors are safe<br />What if a thread close a fd when other thread is block reading it?<br />What happens if a thread add a fd to epoll watch list while other thread is epoll_wait()ing it?<br />What happens if two threads poll same fd, and find it readable simultaneously?<br />What if two threads read the same TCP socket but each get partial data? How do you tell which part comes first?<br />Rule: all operations on one file descriptor should happen in one thread, make your life a lot easier<br />2011/02<br />Shuo Chen (<br />21<br />
  22. 22. File descriptors in threads<br />File descriptors are small integers, unlike HANDLE<br />When create a new fd, kernel picks the lowest unused one<br />Higher possibility of cross-talk, if careless, eg.<br />A fd shared by two threads<br />The first thread have just close()d it<br />The second is about to read() it<br />But a third thread happened to create a new fd with same id (the lowest available int reused) during the period<br />What does the second thread read from? Any other impact?<br />Solution: manage resource with RAII idiom<br />And use the usual technique to manage object life cycles<br />2011/02<br />Shuo Chen (<br />22<br />
  23. 23. C++ and fork()<br />A object could construct once but destruct twice<br />int main()<br />{<br />Foofoo; // call 'Foo::Foo'<br /> fork(); // fork to two process<br />// call 'Foo::~Foo' in parent *and* child processes<br />}<br />It might be a problem, if Foo owns some resource that is not inherited by child process<br />Again, avoid static or global objects in C++<br />In child process, the object may not be properly initialized<br />A global muduo::Timestamp startTime(now()) is wrong<br />2011/02<br />Shuo Chen (<br />23<br />
  24. 24. RAII and fork()<br />fork() doesn't copy all state<br />Open file descriptors are inherited by child process<br />But the offset of file are independent<br />The child does not inherit<br />its parent's memory locks (mlock(2), mlockall(2))<br />record locks from its parent (fcntl(2))<br />timers from its parent (setitimer(2), alarm(2), timer_create(2)), and others<br />So the RAII idiom may not work well in fork()ed process<br />A RAII class that wraps timer_create/timer_delete in ctor/dtor may fail in child process after fork()<br />Use pthread_atfork() as the last resort<br />2011/02<br />Shuo Chen (<br />24<br />
  25. 25. C++ and threads<br />Use scoped lock guard only, check muduo/base/Mutex.h<br />Don't allow exceptions to propagate across module boundaries<br />don't let exception propagate out of the thread main function, catch all exceptions in the outer-most function<br />But, rethrow the one of pthread_cancel(), as we said before<br />Don't allow exceptions to propagate out of your callback, esp. callbacks from C library, eg. the init_routine registered to pthread_once()<br />Better: don't use exception in C++<br />2011/02<br />Shuo Chen (<br />25<br />
  26. 26. Threads and fork()<br />The fork() model doesn’t fit well in threads<br />A fundamental flaw of PosixOSes, as other threads disappear in child, the state is not consistent in child proc<br />After fork a multi-threaded program you may only call async-signal-safe functions in child, as if in signal handler<br />malloc() is not safe, other thread may hold the lock when fork()ing, and no chance to unlock in the new process<br />So does printf(), pthread_* and others.<br />The only safe way to use fork() in a multi-threaded program is calling exec() immediately in child process<br />And make sure set close-on-exec flag on every file descriptors in parent process for security reasons.<br />2011/02<br />Shuo Chen (<br />26<br />
  27. 27. Signals and threads<br />The whole Posix signal mechanism is a shit<br />Only async-signal-safe functions can be called in signal handler, also called 'reentrant functions'<br />Most of the functions are notasync-signal-safe, except those listed in Posix standards, so it's a white list<br /><br />'man 7 signal' to get the list on Linux <br />None of pthread_* are not async-signal-safe, you can't notify a condvar or lock a mutex in signal handler<br />Surprisely, gettimeofday(2) is not async-signal-safe<br />2011/02<br />Shuo Chen (<br />27<br />
  28. 28. Deal with signals in MT programs<br />Rule 1: do not use signal<br />don't use it as IPC, eg. SIGUSR1, SIGUSR2, SIGINT, SIGHUP<br />don't use library functions built upon signals, eg. alarm, sleep, usleep, timer_create, etc.<br />Rule 2: when you absolutely need, convert an async signal to synchronous file descriptor readable event<br />use signalfd in high Linux kernel version<br />Normally, the set of signals to be received via the file descriptor should be blocked using pthread_sigmask(3), to prevent the signals being handled according to their default dispositions.<br />or open a pipe(2), write(2) one byte in signal handler, and read(2) or poll(2) it in main thread<br />2011/02<br />Shuo Chen (<br />28<br />
  29. 29. Other resources<br /><br /><br /><br /><br />Seven posts in<br />2011/02<br />Shuo Chen (<br />29<br />
  30. 30. To be continued<br />Essential of non-blocking network programming in C++<br />Birth of a reactor – design and implementation of Muduo<br />2011/02<br />Shuo Chen (<br />30<br />
  31. 31. Avoid static or global objects<br />Except for PODs<br />2011/02<br />Shuo Chen (<br />31<br />