Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An Introduction to the Formalised Memory Model for Linux Kernel

139 views

Published on

Linux kernel provides executable and formalized memory model. These slides describe the nature of parallel programming in the Linux kernel and what memory model is and why it is necessary and important for kernel programmers. The slides were used at KOSSCON 2018 (https://kosscon.kr/).

Published in: Software
  • Be the first to comment

  • Be the first to like this

An Introduction to the Formalised Memory Model for Linux Kernel

  1. 1. An Introduction to the Formalised Memory Model for Linux Kernel SeongJae Park <sj38.park@gmail.com>
  2. 2. I, SeongJae Park ● SeongJae Park <sj38.park@gmail.com> ● Contributing to the Linux kernel just for fun and profit since 2012 ● Developing GCMA (https://github.com/sjp38/linux.gcma) ● Maintaining Korean translation of Linux kernel memory barrier document ○ The translation has merged into mainline since v4.9-rc1 ○ https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/ko_KR/memory-barriers.txt?h=v4.9-rc1
  3. 3. Programmers in Multi-core Land ● Processor vendors changed their mind to increase number of cores instead of clock speed a decade ago ○ Now, multi-core system is prevalent ○ Even octa-core mobile phone in your pocket, maybe? ● As a result, the free lunch is over; parallel programming is essential for high performance and scalability http://www.gotw.ca/images/CPU.png GIVE UP :’(
  4. 4. Writing Correct Parallel Program is Hard ● Compilers and processors are heavily optimized for Instructions Per Cycle, not programmer perspective goals such as response time or throughput of meaningful (in people’s context) progress ● Nature of parallelism is counter-intuitive ○ Time is relative, before and after is ambiguous, even simultaneous available ● C language developed with Uni-Processor assumption ■ “Et tu, C?” CPU 0 CPU 1 X = 1; Y = 1; X = 2; Y = 2; assert(Y == 2 && X == 1) CPU 1 assertion can be true in Linux Kernel
  5. 5. TL; DR ● Memory operations can be reordered, merged, or discarded in any way unless it violates memory model defined behavior ○ In short, ``We’re all mad here’’ in parallel land ● Knowing memory model is important to write correct, fast, scalable parallel program https://ih1.redbubble.net/image.218483193.6460/sticker,220x200-pad,220x200,ffffff.u2.jpg
  6. 6. Reordering for Better IPC[*] [*] IPC: Instructions Per Cycle
  7. 7. Simple Program Execution Sequence ● Programmer writes program in C-like human readable language ● Compiler translates human readable code into assembly language ● Assembler generates executable binary from the assembly code ● Processor executes instruction sequence in the binary ○ Execution result is guaranteed to be same with sequential execution; In other words, the execution itself is not guaranteed to be sequential #include <stdio.h> int main(void) { printf("hello worldn"); return 0; } main: .LFB0: .cfi_startproc pushq %rbp .cfi_def_cfa_offset 16 .cfi_offset 6, -16 movq %rsp, %rbp 00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000 .ELF............ 00000010: 0200 3e00 0100 0000 3004 4000 0000 0000 ..>.....0.@..... 00000020: 4000 0000 0000 0000 d819 0000 0000 0000 @............... 00000030: 0000 0000 4000 3800 0900 4000 AssemblerCompiler
  8. 8. Instruction Level Parallelism (ILP) ● Pipelining introduces instruction level parallelism ○ Each instruction is splitted up into a sequence of steps; Each step can be executed in parallel, instructions can be processed concurrently fetch decode execute fetch decode execute fetch decode execute If not pipelined, 3 cycles per instruction 3-depth pipeline can retire 3 instructions in 5 cycle: 1.7 cycles per instruction Instruction 1 Instruction 2 Instruction 3
  9. 9. Dependent Instructions Harm ILP ● If an instruction is dependent to result of previous instruction, it should wait until the previous one finishes execution ○ E.g., a = b + c; d = a + b; fetch decode execute fetch decode execute fetch decode execute In this case, instruction 2 depends on result of instruction 1 (e.g., first instruction modifies opcode of next instruction) 7 cycles for 3 instructions: 2.3 cycles per instruction Instruction 1 Instruction 2 Instruction 3
  10. 10. Instruction Reordering Helps Performance ● By reordering dependent instructions to be located in far away, total execution time can be shorten ● If the reordering is guaranteed to not change the result of the instruction sequence, it would be helpful for better performance fetch decode execute fetch decode execute fetch decode execute Instruction 1 Instruction 3 Instruction 2 instruction 2 depends on result of instruction 1 (e.g., first instruction modifies opcode of next instruction) By reordering instruction 2 and 3, total execution time can be shorten 6 cycles for 3 instructions: 2 cycles per instruction
  11. 11. Reordering is Legal, Encouraged Behavior, But... ● If program causality is guaranteed, any reordering is legal ● Processors and compilers can make reordering of instructions for better IPC ● The program causality in C is defined with single processor environment ● IPC focused reordering doesn’t aware programmer perspective performance goals such as throughput or latency ● On Multi-processor system, reordering could harm not only correctness, but also performance
  12. 12. Counter-intuitive Nature of Parallelism
  13. 13. Time is Relative (E = MC2 ) ● Each CPU generates their events in their time, observes effects of events in relative time ● It is impossible to define absolute order of two concurrent events; Only relative observation order is possible CPU 1 CPU 2 CPU 3 CPU 4 Generated event 1 Generated event 2 Observed event 1 followed by event 2 I observed event 2 followed by event 1 Event Bus
  14. 14. Relative Event Propagation of Hierarchical Memory ● Most system equip hierarchical memory for better performance and space ● Propagation speed of an event to a given core can be influenced by specific sub-layer of memory If CPU 0 Message Queue is busy, CPU 2 can observe an event from CPU 0 (event A) after an event of CPU 1 (event B) though CPU 1 observed event A before generating event B CPU 0 CPU 1 Cache CPU 0 Message Queue CPU 1 Message Queue Memory CPU 2 CPU 3 Cache CPU 2 Message Queue CPU 3 Message Queue Bus
  15. 15. Relative Event Propagation of Hierarchical Memory ● Most system equip hierarchical memory for better performance and space ● Propagation speed of an event to a given core can be influenced by specific sub-layer of memory If CPU 0 Message Queue is busy, CPU 2 can observe an event from CPU 0 (event A) after an event of CPU 1 (event B) though CPU 1 observed event A before generating event B CPU 0 CPU 1 Cache CPU 0 Message Queue CPU 1 Message Queue Memory CPU 2 CPU 3 Cache CPU 2 Message Queue CPU 3 Message Queue Bus Generate Event A; Event A
  16. 16. Relative Event Propagation of Hierarchical Memory ● Most system equip hierarchical memory for better performance and space ● Propagation speed of an event to a given core can be influenced by specific sub-layer of memory If CPU 0 Message Queue is busy, CPU 2 can observe an event from CPU 0 (event A) after an event of CPU 1 (event B) though CPU 1 observed event A before generating event B CPU 0 CPU 1 Cache CPU 0 Message Queue CPU 1 Message Queue Memory CPU 2 CPU 3 Cache CPU 2 Message Queue CPU 3 Message Queue Bus Generate Event A; Seen Event A; Generate Event B; Event BEvent A
  17. 17. Relative Event Propagation of Hierarchical Memory ● Most system equip hierarchical memory for better performance and space ● Propagation speed of an event to a given core can be influenced by specific sub-layer of memory If CPU 0 Message Queue is busy, CPU 2 can observe an event from CPU 0 (event A) after an event of CPU 1 (event B) though CPU 1 observed event A before generating event B CPU 0 CPU 1 Cache CPU 0 Message Queue CPU 1 Message Queue Memory CPU 2 CPU 3 Cache CPU 2 Message Queue CPU 3 Message Queue Bus Generate Event A; Seen Event A; Generate Event B; Seen Event B; Event A Event B Busy… ;;;
  18. 18. Relative Event Propagation of Hierarchical Memory ● Most system equip hierarchical memory for better performance and space ● Propagation speed of an event to a given core can be influenced by specific sub-layer of memory If CPU 0 Message Queue is busy, CPU 2 can observe an event from CPU 0 (event A) after an event of CPU 1 (event B) though CPU 1 observed event A before generating event B CPU 0 CPU 1 Cache CPU 0 Message Queue CPU 1 Message Queue Memory CPU 2 CPU 3 Cache CPU 2 Message Queue CPU 3 Message Queue Bus Generate Event A; Seen Event A; Generate Event B; Seen Event B; Seen Event A; Event B Event A
  19. 19. Cache Coherency is Per-CacheLine ● It is well known that cache coherency protocol helps system memory consistency ● In actual, it guarantees sequential consistency, but per-cache-line only ● Every CPU will eventually agree about global order of each cache-line, but some CPU can aware the change faster than others, order of changes between different cache-lines is not guaranteed http://img06.deviantart.net/0663/i/2005/112/b/6/schrodinger__s_cat___2_by_firefoxcentral.jpg
  20. 20. System with Store Buffer and Invalidation Queue ● Store Buffer and Invalidation Queue deliver effect of event but does not guarantee order of observation on each CPU CPU 0 Cache Store Buffer Invalidation Queue Memory CPU 1 Cache Store Buffer Invalidation Queue Bus
  21. 21. C-language and Multi-Processor
  22. 22. C-language Doesn’t Know Multi-Processor ● By the time of initial C-language development, multi-processor was rare ● As a result, C-language has only few guarantees about memory operations on multi-processor ● Undefined behavior is allowed for undefined case https://upload.wikimedia.org/wikipedia/commons/thumb/9/95/The_C_Programming _Language,_First_Edition_Cover_(2).svg/2000px-The_C_Programming_Languag e,_First_Edition_Cover_(2).svg.png
  23. 23. Compiler Optimizes Code ● Clever compilers try hard (really hard) to optimize code for high IPC (again, not for programmer perspective goals) ○ Converts small, private function to inline code ○ Reorder memory access code to minimize dependency ○ Simplify unnecessarily complex loops, ... ● Optimization uses term `Undefined behavior` as they want ○ It’s legal, but sometimes do insane things in programmer’s perspective ● Memory access reordering of compiler based on C-standard, which doesn’t aware multi-processor system, can generate unintended program ● Linux kernel uses compiler directives and volatile keyword to enforce memory ordering ● C11 has much more improvement, though
  24. 24. Memory Models
  25. 25. Memory Consistency Model (a.k.a Memory Model) ● Defines what values can be obtained by the code’s load instructions ● Each programming environment including Instruction Set Architecture, Programming language, Operating system defines own memory model ○ Modern language memory models (e.g., Golang, Rust, Java, C11, …) aware multi-processor http://www.sciencemag.org/sites/default/files/styles/article_main_large/public/Memory.jpg?itok=4FmHo7M5
  26. 26. Each ISA Provides Specific Memory Model ● Some architectures have stricter ordering enforcement rule than others ● PA-RISC CPUs are strictest, Alpha is weakest ● Because Linux kernel supports multiple architectures, it defines its memory model based on weakest one, the Alpha https://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.2015.01.31a.pdf
  27. 27. Synchronization Primitives ● Because reordering and asynchronous effect propagation is legal, synchronization primitives are necessary to write human intuitive program ● Most memory model provides synchronization primitives including atomic read-modify-write instructions and memory barriers. https://s-media-cache-ak0.pinimg.com/236x/42/bc/55/42bc55a6d7e5affe2d0dbe9c872a3df9.jpg
  28. 28. Atomic Operations ● Atomic operations are configured with multiple sub operations ○ E.g., compare-and-swap, fetch-and-add, test-and-set ● Atomic operations have mutual exclusiveness ○ Middle state of atomic operation execution cannot be seen by others ○ It can be thought of as small critical section that protected by a global lock ● Almost every hardware supports basic atomic operations http://www.scienceclarified.com/photos/atomic-mass-3024.jpg
  29. 29. Memory Barriers ● To allow synchronization of memory operations, memory model provides enforcement primitives, namely, memory barriers ● In general, memory barriers guarantee effects of memory operations issued before it to be propagated to other components (e.g., processor) in the system before memory operations issued after the barrier CPU 1 CPU 2 CPU 3 READ A; WRITE B; <BARRIER>; READ C; READ A, WRITE B, than READ C occurred WRITE B, READ A, than READ C occurred READ A and WRITE B can be reordered but READ C is guaranteed to be ordered after {READ A, WRITE B}
  30. 30. Cost of Synchronization Primitives ● Generally, synchronization primitives are expensive, unscalable ● Performance between synchronization primitives are different, though ● Correct selection and efficient use of synchronization primitives are important! Performance of various synchronization primitives
  31. 31. LKMM: Linux Kernel Memory Model
  32. 32. Linux Kernel Memory Model ● The memory model for Linux kernel programming environment ○ Defines what values can be obtained, given a piece of Linux kernel code, for specific load instructions in the code ● Linux kernel original memory model is necessary because ○ It uses C99 but C99 memory model is oblivious of multi-processor environment; C11 memory model aware of multi-processor, but Torvalds doesn’t want it ○ It supports multiple architectures with different ISA-level memory model https://github.com/sjp38/perfbook
  33. 33. LKMM: Linux Kernel Memory Model ● Designed for weakest memory model architecture, Alpha ○ Almost every combination of reordering is possible, doesn’t provide address dependency ● Atomic instructions ○ atomix_xchg(), atomic_inc_return(), atomic_dec_return(), … ○ Most machines have counterpart instructions, but kernel atomic RMW primitives provide general guarantees at Kernel level ● Memory barriers ○ Compiler barriers: WRITE_ONCE(), READ_ONCE(), barrier(), ... ○ CPU barriers: mb(), wmb(), rmb(), smp_mb(), smp_wmb(), smp_rmb(), … ○ Semantic barriers: ACQUIRE operations, RELEASE operations, … ○ For detail, refer to https://www.kernel.org/doc/Documentation/memory-barriers.txt ● Because different memory ordering primitive has different cost, only necessary ordering primitives should be used in necessary case for high performance and scalability
  34. 34. Informal LKMM ● Originally, LKMM was just an informal text, ‘memory-barriers.txt’ ● It explains about the Linux Kernel Memory Model in English (There is Korean translation, too: https://www.kernel.org/doc/Documentation/translations/ko_KR/memory-barriers.txt) ● To use the LKMM to prove your code, you should use Feynman Algorithm ○ Write down your code ○ Think real hard with the ‘memory-barriers.txt’ ○ Write down your provement ○ (Hard and unreliable, of course!) https://www.kernel.org/doc/Documentation/memory-barriers.txt
  35. 35. Formal LKMM: Help Arrives at Last ● It is formal, executable memory model ○ It receives C-like simple code as input ○ The code containing parallel code snippets and a question: can this result happen? ● Based on herd7 and klitmus7 ○ LKMM extends herd7 and klitmus7 to support LKMM ordering primitives in code ○ Herd7 simulates in user mode, klitmus7 runs in real kernel mode ● Few limitations exist, of course https://i.pinimg.com/originals/a9/5f/cd/a95fcd3519fe3222f07d59b0c1536305.png
  36. 36. LKMM Demonstration ● Installation ○ LKMM is merged in Linux source tree at tools/memory-model; Just pull the linux source code ○ Install herdtools7 (https://github.com/herd/herdtools7) ● Usage ○ Using herd7 user mode simulation $ herd7 -conf linux-kernel.cfg <your-litmus-test file> ○ Using klitmus7 based real kernel mode execution $ mkdir mymodules $ klitmus7 -o mymodules <your-litmus-test file> $ cd mymodules ; make $ sudo sh run.sh ● That’s it! Now you can prove your parallel code for all Linux environments!
  37. 37. Summary ● Nature of Parallel Land is counter-intuitive ○ Cannot define order of events without interaction ○ Ordering rule is different for different environment ○ Memory model defines their ordering rule ○ In short, they’re all mad here ● For human-intuitive and correct program, interaction is necessary ○ Almost every environment provides memory ordering primitives including atomic instructions and memory barriers, which is expensive in common ○ Memory model defines what result can occur and cannot with given code snippet ● Formal Linux kernel memory model is available ○ Linux kernel provides its memory model based on weakest memory ordering rule architecture it supports, the Alpha, C99, and its original ordering primitives including RCU ○ Formal LKMM using herd7 is merged to mainstream; now you can prove your parallel code!
  38. 38. Thanks
  39. 39. This work by SeongJae Park is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/.
  40. 40. Case Studies
  41. 41. Memory Operation Reordering ● Memory Operation Reordering is totally LEGAL unless it breaks causality ● Both of CPU and Compiler can do it, even in Single Processor CPU 0 CPU 1 CPU 2 A = 1; B = 1; while (B == 0) {} C = 1; Z = C; X = A; assert(z == 0 || x == 1)
  42. 42. Memory Operation Reordering ● Memory Operation Reordering is totally LEGAL unless it breaks causality ● Both of CPU and Compiler can do it, even in Single Processor CPU 0 CPU 1 CPU 2 A = 1; B = 1; while (B == 0) {} C = 1; Z = C; X = A; assert(z == 0 || x == 1) :)
  43. 43. Memory Operation Reordering ● Memory Operation Reordering is totally LEGAL unless it breaks causality ● Both of CPU and Compiler can do it, even in Single Processor CPU 0 CPU 1 CPU 2 A = 1; B = 1; while (B == 1) {} C = 1; Z = C; X = A; assert(z == 0 || x == 1) :)
  44. 44. Memory Operation Reordering ● Memory Operation Reordering is totally LEGAL unless it breaks causality ● Both of CPU and Compiler can do it, even in Single Processor CPU 0 CPU 1 CPU 2 B = 1; A = 1; while (B == 0) {} C = 1; Z = C; X = A; assert(z == 0 || x == 1)
  45. 45. Memory Operation Reordering ● Memory Operation Reordering is totally LEGAL unless it breaks causality ● Both of CPU and Compiler can do it, even in Single Processor CPU 0 CPU 1 CPU 2 B = 1; A = 1; while (B == 0) {} C = 1; X = A; Z = C; assert(z == 0 || x == 1) ?????
  46. 46. Memory Operation Reordering ● Memory Operation Reordering is totally LEGAL unless it breaks causality ● Both of CPU and Compiler can do it, even in Single Processor ● Memory barrier enforces operations specified before it appear as happened to operations specified after it CPU 0 CPU 1 CPU 2 A = 1; wmb(); B = 1; while (B == 0) {} mb(); C = 1; Z = C; rmb(); X = A; assert(z == 0 || x == 1)
  47. 47. Memory Operation Reordering ● Memory Operation Reordering is totally LEGAL unless it breaks causality ● Both of CPU and Compiler can do it, even in Single Processor ● Memory barrier enforces operations specified before it appear as happened to operations specified after it ● In some architecture, even Transitivity is not guaranteed ○ Transitivity: B happened after A; C happened after B; then C happened after A CPU 0 CPU 1 CPU 2 A = 1; wmb(); B = 1; while (B == 0) {} mb(); C = 1; Z = C; rmb(); X = A; assert(z == 0 || x == 1)
  48. 48. Transitivity for Scheduler and Workers Scheduler and each workers made consensus about order Scheduler Worker A Worker B Worker Z ... ... What time is it now? Night! Night! Night! ...
  49. 49. Transitivity between Scheduler and Worker Scheduler and each workers made consensus about order Scheduler Worker A Worker B Worker Z ... ... Yay! ... Worker Z, all workers agreed that it’s night. Do bedmaking!
  50. 50. Transitivity between Scheduler and Worker Scheduler and each workers made consensus about order But, worker B and worker Z didn’t made consensus Scheduler Worker A Worker B Worker Z ... ... !!?? ... Worker Z, I’m in afternoon! I didn’t tell you it’s night!
  51. 51. Compiler Memory Barrier Code Example
  52. 52. Compiler Reordering Avoidance ● Compiler can remove loop entirely C code Assembly language code static int the_var; void loop(void) { int i; for (i = 0; i < 1000; i++) the_var++; } loop: .LFB106: .cfi_startproc addl $1000, the_var(%rip) ret .cfi_endproc .LFE106:
  53. 53. Compiler Reordering Avoidance ● ACCESS_ONCE() is a compiler memory barrier implementation of Linux kernel ● Store to the_var could not be seen by others C code Assembly language code static int the_var; void loop(void) { int i; for (i = 0; ACCESS_ONCE(i) < 1000; i++) the_var++; } loop: ... movl the_var(%rip), %ecx .L175: ... addl $1, %eax ... cmpl $999, %edx jle .L175 movl %esi, the_var(%rip) .L170: rep ret
  54. 54. Compiler Reordering Avoidance ● Still, store to `the_var` not issued for every iteration C code Assembly language code static int the_var; void loop(void) { int i; for (i = 0; ACCESS_ONCE(i) < 1000; i++) the_var++; } loop: ... movl the_var(%rip), %ecx .L175: ... addl $1, %eax ... cmpl $999, %edx jle .L175 movl %esi, the_var(%rip) .L170: rep ret
  55. 55. Compiler Reordering Avoidance ● volatile enforces compiler to issue memory operation as programmer want (Note that it is not enforced to do DRAM access) ● However, repetitive LOAD may harm performance C code Assembly language code static volatile int the_var; void loop(void) { int i; for (i = 0; ACCESS_ONCE(i) < 1000; i++) the_var++; } loop: ... .L174: movl the_var(%rip), %edx ... addl $1, %edx movl %edx, the_var(%rip) ... cmpl $999, %edx jle .L174 .L170: rep ret .cfi_endproc
  56. 56. Compiler Reordering Avoidance ● Complete memory barrier can help the case ● Does memory access once and uses register for loop condition check C code Assembly language code static int the_var; void loop(void) { int i; for (i = 0; i < 1000; i++) the_var++; barrier(); } loop: .LFB106: ... .L172: addl $1, the_var(%rip) subl $1, %eax jne .L172 rep ret .cfi_endproc
  57. 57. CPU Memory Barrier Code Example
  58. 58. Progress perception ● Code does issue LOAD and STORE, but… ● see_progress() can see no progress because change made by a processor propagates to other processor eventually, not immediately C code Assembly language code static int prgrs; void do_progress(void) { prgrs++; } void see_progress(void) { static int last_prgrs; static int seen; static int nr_seen; seen = prgrs; if (seen > last_prgrs) nr_seen++; last_prgrs = seen; } do_progress: ... addl $1, prgrs(%rip) ret ... see_progress: ... movl prgrs(%rip), %eax ... jle .L193 addl $1, nr_seen.5542(%rip) .L193: movl %eax, last_prgrs.5540(%rip) ret .cfi_endproc
  59. 59. Progress perception ● Read barrier and write barrier helps the situation C code Assembly language code static int prgrs; void do_progress(void) { prgrs++; smp_wmb(); } void see_progress(void) { static int last_prgrs; static int seen; static int nr_seen; smp_rmb(); seen = prgrs; if (seen > last_prgrs) nr_seen++; last_prgrs = seen; } do_progress: ... addl $1, prgrs(%rip) ... sfence ret see_progress: ... lfence ... movl prgrs(%rip), %eax ... jle .L193 addl $1, nr_seen.5542(%rip) .L193: movl %eax, last_prgrs.5540(%rip)
  60. 60. Memory Ordering of X86
  61. 61. Neither Loads Nor Stores Are Reordered with Likes CPU 0 CPU 1 STORE 1 X STORE 1 Y R1 = LOAD Y R2 = LOAD X R1 == 1 && R2 == 0 impossible
  62. 62. Stores Are Not Reordered With Earlier Loads CPU 0 CPU 1 R1 = LOAD X STORE 1 Y R2 = LOAD Y STORE 1 X R1 == 1 && R2 == 1 impossible
  63. 63. Loads May Be Reordered with Earlier Stores to Different Locations CPU 0 CPU 1 STORE 1 X R1 = LOAD Y STORE 1 Y R2 = LOAD X R1 == 0 && R2 == 0 possible
  64. 64. Intra-Processor Forwarding Is Allowed CPU 0 CPU 1 STORE 1 X R1 = LOAD X R2 = LOAD Y STORE 1 Y R3 = LOAD Y R4 = LOAD X R2 == 0 && R4 == 0 possible
  65. 65. Stores Are Transitively Visible CPU 0 CPU 1 CPU 2 STORE 1 X R1 = LOAD X STORE 1 Y R2 = LOAD Y R3 = LOAD X R1 == 1 && R2 == 1 && R3 == 0 impossible
  66. 66. Stores Are Seen in a Consistent Order by Others CPU 0 CPU 1 CPU 2 CPU 3 STORE 1 X STORE 1 Y R1 = LOAD X R2 = LOAD Y R3 = LOAD Y R4 = LOAD X R1 == 0 && R2 == 0 && R3 == 1 && R4 == 0 impossible
  67. 67. X86 Memory Ordering Summary ● LOAD after LOAD never reordered ● STORE after STORE never reordered ● STORE after LOAD never reordered ● STOREs are transitively visible ● STOREs are seen in consistent order by others ● Intra-processor STORE forwarding is possible ● LOAD from different location after STORE may be reordered ● In short, quite reasonably strong enough ● For more detail, refer to `Intel Architecture Software Developer’s Manual`
  68. 68. Summary ● Nature of Parallel Land is counter-intuitive ○ Cannot define order of events without interaction ○ Ordering rule is different for different environment ○ Memory model defines their ordering rule ○ In short, they’re all mad here ● For human-intuitive and correct program, interaction is necessary ○ Every memory model provides synchronization primitives like atomic instruction and memory barrier, etc ○ Such interaction is expensive in common ● Linux kernel memory model is based on weakest memory model, Alpha ○ Kernel programmers should assume Alpha when writing architecture independent code ○ Because of the expensive cost of synchronization primitives, programmer should use only necessary primitives on necessary location

×