This document discusses challenges in multithreaded system programming in C++. It covers topics such as thread safety of libraries, RAII and fork(), signals and threads, and operating file descriptors in threads. The document is intended for C++ programmers familiar with threads and aims to explain interactions between threads and system calls/libraries to avoid common issues.

1. Essentials of MultithreadedSystem Programming in C++Shuo Chen2011/02blog.csdn.net/Solsticegiantchen@gmail.com@bnu_chenshuo

2. ContentsChallenges in multithreaded system programmingThread safety of C and C++ librariesRAII and fork()fork() and signal handling in multithreaded programs2011/02Shuo Chen (blog.csdn.net/Solstice)2

3. Audience: C++ programmersFamiliar with Pthreads and Sockets APIKnows thread safety, deadlock, race condition, etc.In a word: read through APUE2e and UNP3e (vol. 1) by W. Richard Stevens et al.All discussions are based on Linux 2.6.x, x >= 28There are new syscalls, egsignalfd, eventfd, and timerfdx86 and x64 platforms2011/02Shuo Chen (blog.csdn.net/Solstice)3

4. Multi-threaded system programmingMultithreading is inevitable in this multi-core eraThe difficulties are not learning synchronization primitives (mutexes, condition variables)~10 functions are sufficient to do it rightBut understanding interactions between existing system calls and library functionsUnderstands how threads affect system designUse it wisely and effectivelyAvoid common pitfalls and fallacies2011/02Shuo Chen (blog.csdn.net/Solstice)4

5. 11 essential Pthreads functions11 out of 110+ pthreads functions2 -> create and join threads4 -> init/destroy, lock/unlock mutexes5 -> init/destroy, wait/signal/broadcast condvarsThink twice if you need moreSome are okay, eg. once and key, maybe rwlockSome are bad, eg. cancel and kill, semaphoresCheck muduo/base for encapsulation in C++http://code.google.com/p/muduohttp://github.com/chenshuo/recipes click thread2011/02Shuo Chen (blog.csdn.net/Solstice)5

6. An asynchronous worldNever assume the sequence of events without proper synchronization.Knows happens-before relation, memory visibility, etc.The effect of an interaction between two [thread]s must be independent of the speed at which it is carried out. --- Brinch Hansen 19732011/02Shuo Chen (blog.csdn.net/Solstice)6

7. Standards and practicesAlthough the latest official standards of C and C++ languages (C99 and C++03) do not say a word about process or threadWe write multi-process and/or multi-threaded C/C++ programs in real life, as a real-world needWe can’t wait it to be standardized, as standards usually fall behind practices for yearsbtw, if there are not real life multi-threads programs , how do people what/how to standardize?We adhere to some de facto standardsA lot simpler if we focus on one hardware and one OS2011/02Shuo Chen (blog.csdn.net/Solstice)7

8. Thread identifier on LinuxUse pid_t as thread id, instead of pthread_t, on Linuxpthread_tthid = pthread_self(), thid is opaque (uintptr_t)pid_ttid = ::gettid(), tid is task id, usually a small integer/proc/tid/, /proc/pid/task/tid/, ps, top all work fineHow to implement gettid() efficiently? Thread local?gettid(2) is a syscall, but the output should never changegetpid(2) caches the result, should gettid() do the same?What if fork(), will it caches the old value in child proc?How about pthread_atfork() to clear it up?Check muduo/base/Thread.cc for details2011/02Shuo Chen (blog.csdn.net/Solstice)8

9. Creation of threadsA library should not create its own ‘background’ thread without prior informed consentMakes a program non-forkableNever create thread before main()Avoid creating thread in ctor of static or global objectBreaks static objects constructing, eg. protobuf registeringThe number of threads created should be independentof system load, eg. # of connections, # of requestsotherwise non-scalableReuse threads, by assign multiple roles to itDoing IO and timer with muduoEventLoop classFor simple task, do it within IO callbacks in IO threads2011/02Shuo Chen (blog.csdn.net/Solstice)9

10. Three ways of terminationhttp://blog.csdn.net/program_think/archive/2009/03/14/3991107.aspxNatural death – return from thread function, goodSuicide – call pthread_exit()Mudered – killed by pthread_cancel()Rule: let it die, never suicide or murder a threadWhy? inherently deadlock-prone: no chance to unlockDesign your program so that a thread can be waken up and safely exitsFor referenceJava Thread.{stop, suspend and destroy} are deprecatedBoost Threads doesn’t provide thread::cancel()2011/02Shuo Chen (blog.csdn.net/Solstice)10

11. pthread_cancel() and C++In C, we have concept of ‘cancellation point’In C++, pthread_cancel() throws an exception in that thread, helps unwinding objects on stackThe exception must reach the outmost function, otherwise core dump:FATAL: exception not rethrownAborted (core dumped)Always rethrow in catch(…) causeUlrich Drepper “Cancellation and C++ Exceptions”Better: never cancel or kill a thread2011/02Shuo Chen (blog.csdn.net/Solstice)11

12. exit() is not thread safe in C++exit() destructs static or global objects, (_exit() doesn’t)The destructor may try to hold a lockThe caller function may have held the same lock alreadyEnd up in a dead lockCheck following code for an example of dead lock github.com/chenshuo/recipes/blob/master/thread/test/ExitDeadLock.ccHow to quit a multi-threaded program safely?An irregular but simple solution: make a process killable, eg.p.29 blog.csdn.net/Solstice/archive/2010/10/19/5950190.aspxIt’s not fault of exit(), but static or global objectsTry to avoid static or global objects in C++, except for PODs2011/02Shuo Chen (blog.csdn.net/Solstice)12

13. Thread local __thread in g++Thread safe by natural, unless escaped to other threadMore efficient implementation, than pthread_key_tSee “ELF Handling For Thread-Local Storage”In C++, must be initialized with constant-expressionNo __thread string t_obj("Chen Shuo");No __thread string* t_obj = new string;Only __thread string* t_obj = NULL;More rules: http://gcc.gnu.org/onlinedocs/gcc/Thread_002dLocal.htmlUse pthread_key_t if you want auto destruction2011/02Shuo Chen (blog.csdn.net/Solstice)13

14. Use non-recursive mutex onlyA basic assumption of holding a mutexOnce I lock it, I can modify the guarded object safelyWhich is not true for recursive mutex, eg.http://blog.csdn.net/Solstice/archive/2010/02/12/5307710.aspx#_Toc11928Recursive mutexes by David Butenhofhttp://zaval.org/resources/library/butenhof1.html Recursive locks - a blessing or a curse? http://www.thinkingparallel.com/2006/09/27/recursive-locks-a-blessing-or-a-curse/2011/02Shuo Chen (blog.csdn.net/Solstice)14

15. Impacts of introducing threadsThreading is a late patch to OS kernelUnix kernel and API formed in early 1970sFirst implementation of threads emerged in early 1990sBreaks lots of assumptions made during the 20 yearsLibrary functions with side effects must be revisitedmalloc/free, fread/fseekcan be made thread-safe with locksFunctions that return or use static allocated space are not thread safe but may have thread-safe variantsasctime_r, ctime_r, gmtime_r, rand_r, stderror_r, strtok_rerrno is not an ‘extern int’, but a per-thread valueextern int *__errno_location(void);#define errno (*__errno_location())2011/02Shuo Chen (blog.csdn.net/Solstice)15

16. Thread safety of C libraryIndividual system calls must be thread safeBe caution of interfering of same file descriptor from multiple threadsMost of glibc library functions are thread safe nowadaysCounterintuitively, Posix standards lists functions thatare not required to be thread safe, it's a black list.http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_092.9.1 Thread-Safety :All functions defined by this volume of POSIX.1-2008 shall be thread-safe, except that the following functions need not be thread-safe.Notably, getenv/putenv/setenv/system() are not safe2011/02Shuo Chen (blog.csdn.net/Solstice)16

17. FILE* functions are thread safeRead ‘man flockfile’, but they are not composable, eg.fseek(), followed by fread()The file position may change during the course by a different threadWrap with flockfile(FILE*) and funlockfile(FILE*) Same applies to lseek(2) and read(2), but how to lock?Use pread(2) instead, which doesn’t change the file offsetIn general, a function that calls two thread-safe functions is not guaranteed to be thread-safeJust like exception-safety, thread-safety is not composable2011/02Shuo Chen (blog.csdn.net/Solstice)17

18. Thread safety is not composableA solution works in single-threaded program may not apply to multi-threaded program.Any solution calls two or more thread safe function are not necessarily correct in multi-threaded programWhat’s the time in London now? Program runs in New Yorkstring oldTz = getenv("TZ"); // save TZputenv("TZ=Europe/London"); tzset(); // set TZ to Londonstruct tm localTimeInLN = *localtime(time(NULL));setenv("TZ", oldTz.c_str(), 1); tzset(); // restore old TZThis code impacts localtime() in other threadsThread safe functions are not composable unless you carefully design the interface and interactions2011/02Shuo Chen (blog.csdn.net/Solstice)18

19. Thread safety of C++ std libraryAlthough not required by the standard, the de facto saysUnshared objects are independent: Two threads can freely use different objects without any special action on the caller's part. We call it "same level as built-in types."This applies to STL containers like map, vector, stringPure functions are safe, eg. Most of STL algorithms.The global cin/cout objects are shared by threads, and are not thread safe. Moreover, they can't be made safecout << a << b;  cout.operator<<(a).operator<<(b);Two function calls can be interrupted by another threadUse printf(3) instead, it's thread safe and atomic.Allocators must be thread safe, as they are shared2011/02Shuo Chen (blog.csdn.net/Solstice)19

20. Thread-Safe vs. Thread-Efficientprintf(3) and malloc(3) are thread safe, but not necessarily efficient enough, esp. on multi-coresprintf(3) locks FILE* stdout, synchronizes threadsnot good for multi-threaded logging, we need a better libyour default malloc(3) may not optimized for multi-threads and multi-coresit may lock global heap for each allocationtry tcmalloc, Google's thread-cache mallocsee Intel. Is your memory management multi-core ready?http://software.intel.com/en-us/blogs/2009/08/21/is-your-memory-management-multi-core-ready/2011/02Shuo Chen (blog.csdn.net/Solstice)20

21. Operate one fd in one threadAlthough system calls of file descriptors are safeWhat if a thread close a fd when other thread is block reading it?What happens if a thread add a fd to epoll watch list while other thread is epoll_wait()ing it?What happens if two threads poll same fd, and find it readable simultaneously?What if two threads read the same TCP socket but each get partial data? How do you tell which part comes first?Rule: all operations on one file descriptor should happen in one thread, make your life a lot easier2011/02Shuo Chen (blog.csdn.net/Solstice)21

22. File descriptors in threadsFile descriptors are small integers, unlike HANDLEWhen create a new fd, kernel picks the lowest unused oneHigher possibility of cross-talk, if careless, eg.A fd shared by two threadsThe first thread have just close()d itThe second is about to read() itBut a third thread happened to create a new fd with same id (the lowest available int reused) during the periodWhat does the second thread read from? Any other impact?Solution: manage resource with RAII idiomAnd use the usual technique to manage object life cycles2011/02Shuo Chen (blog.csdn.net/Solstice)22

23. C++ and fork()A object could construct once but destruct twiceint main(){Foofoo; // call 'Foo::Foo' fork(); // fork to two process// call 'Foo::~Foo' in parent *and* child processes}It might be a problem, if Foo owns some resource that is not inherited by child processAgain, avoid static or global objects in C++In child process, the object may not be properly initializedA global muduo::Timestamp startTime(now()) is wrong2011/02Shuo Chen (blog.csdn.net/Solstice)23

24. RAII and fork()fork() doesn't copy all stateOpen file descriptors are inherited by child processBut the offset of file are independentThe child does not inheritits parent's memory locks (mlock(2), mlockall(2))record locks from its parent (fcntl(2))timers from its parent (setitimer(2), alarm(2), timer_create(2)), and othersSo the RAII idiom may not work well in fork()ed processA RAII class that wraps timer_create/timer_delete in ctor/dtor may fail in child process after fork()Use pthread_atfork() as the last resort2011/02Shuo Chen (blog.csdn.net/Solstice)24

25. C++ and threadsUse scoped lock guard only, check muduo/base/Mutex.hDon't allow exceptions to propagate across module boundariesdon't let exception propagate out of the thread main function, catch all exceptions in the outer-most functionBut, rethrow the one of pthread_cancel(), as we said beforeDon't allow exceptions to propagate out of your callback, esp. callbacks from C library, eg. the init_routine registered to pthread_once()Better: don't use exception in C++2011/02Shuo Chen (blog.csdn.net/Solstice)25

26. Threads and fork()The fork() model doesn’t fit well in threadsA fundamental flaw of PosixOSes, as other threads disappear in child, the state is not consistent in child procAfter fork a multi-threaded program you may only call async-signal-safe functions in child, as if in signal handlermalloc() is not safe, other thread may hold the lock when fork()ing, and no chance to unlock in the new processSo does printf(), pthread_* and others.The only safe way to use fork() in a multi-threaded program is calling exec() immediately in child processAnd make sure set close-on-exec flag on every file descriptors in parent process for security reasons.2011/02Shuo Chen (blog.csdn.net/Solstice)26

27. Signals and threadsThe whole Posix signal mechanism is a shitOnly async-signal-safe functions can be called in signal handler, also called 'reentrant functions'Most of the functions are notasync-signal-safe, except those listed in Posix standards, so it's a white listhttp://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_04_03_03'man 7 signal' to get the list on Linux None of pthread_* are not async-signal-safe, you can't notify a condvar or lock a mutex in signal handlerSurprisely, gettimeofday(2) is not async-signal-safe2011/02Shuo Chen (blog.csdn.net/Solstice)27

28. Deal with signals in MT programsRule 1: do not use signaldon't use it as IPC, eg. SIGUSR1, SIGUSR2, SIGINT, SIGHUPdon't use library functions built upon signals, eg. alarm, sleep, usleep, timer_create, etc.Rule 2: when you absolutely need, convert an async signal to synchronous file descriptor readable eventuse signalfd in high Linux kernel versionNormally, the set of signals to be received via the file descriptor should be blocked using pthread_sigmask(3), to prevent the signals being handled according to their default dispositions.or open a pipe(2), write(2) one byte in signal handler, and read(2) or poll(2) it in main thread2011/02Shuo Chen (blog.csdn.net/Solstice)28

29. Other resourceshttp://pubs.opengroup.org/onlinepubs/9699919799/http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-themhttp://www.linuxprogrammingblog.com/all-about-linux-signalshttp://www.cppblog.com/lymons/archive/2008/06/01/51838.htmlSeven posts in http://www.cppblog.com/lymons/category/9446.html2011/02Shuo Chen (blog.csdn.net/Solstice)29

30. To be continuedEssential of non-blocking network programming in C++Birth of a reactor – design and implementation of Muduo2011/02Shuo Chen (blog.csdn.net/Solstice)30

31. Avoid static or global objectsExcept for PODs2011/02Shuo Chen (blog.csdn.net/Solstice)31