A description of Flush+Fluash, Flush+Reload, Meltdown and Spectre attacks and a possible solutions without KPTI .

1. Preventing Preventing
CPU side­channel CPU side­channel
attacks with attacks with
kernel trackingkernel tracking
Marian MarinovMarian Marinov
mm@siteground.commm@siteground.com
Chief System ArchitectChief System Architect
Head of the DevOps departmentHead of the DevOps department

2. ❖❖ Who am I?Who am I?
- Chief System Architect of SiteGround.com- Chief System Architect of SiteGround.com
- Sysadmin since 1996- Sysadmin since 1996
- Organizer of OpenFest, BG Perl- Organizer of OpenFest, BG Perl
Workshops, LUG-BG and othersWorkshops, LUG-BG and others
- Teaching Network Security and- Teaching Network Security and
Linux System AdministrationLinux System Administration
courses in Sofia Universitycourses in Sofia University
and SoftUniand SoftUni

3. There are There are 44**
different different
caches inside the CPUcaches inside the CPU
L1
instructions cache
L2 cache
L3 cache
L1 I
cache
L1 D
cache
L2 cache
L1 I
cache
L1 D
cache
L2 cache
L1
data cache
L1 I
cache
L2
Single Core Single CoreSingle Core Single C

4. There are There are 44**
different different
caches inside the CPUcaches inside the CPU
L1
instructions cache
L2 cache
L3 cache
L1 I
cache
L1 D
cache
L2 cache
L1 I
cache
L1 D
cache
L2 cache
L1
data cache
L1 I
cache
L2
Single Core Single CoreSingle Core Single C
* In some architectures, there is even L4 cache

5. L1 and L2 caches are shared between
hyper-threads in a single core
L2 cache is shared between different
execution engines inside the core
(ALU, FMA, ADD, etc.)
L3 cache is shared between all cores
Sharing the cacheSharing the cache

6. Shared L3 Cache (LLC)
Synchronization
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
L1
Instruction
cache
Isnt.
Instruct
Di
W.C
L1
Instruction
cache
Integer
Cluster
1
L1
data
cache
LCore
Iface
Some CPU architecture intro :)
AMD Bulldozer block diagram

7. Cache Side­Channel Attacks
➢ Flush + Reload
➢ Flush + Flush

8. Flush + ReloadFlush + Reload
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
Shared L3 Cache (LLC)
Synchronization
1. Find a shared library location in memory
2. Clear the cache
3. Check if it is accessed or not by comparing
the time it takes to execute the code

9. Flush + ReloadFlush + Reload
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
Shared L3 Cache (LLC)
Synchronization

10. Flush + ReloadFlush + Reload
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
Shared L3 Cache (LLC)
Synchronization

11. Flush + FlushFlush + Flush
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
Shared L3 Cache (LLC)
Synchronization
1. Find a shared library location in memory
2. Clear the cache
3. Clear the cache again and observe the timing

12. Flush + FlushFlush + Flush
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
Shared L3 Cache (LLC)
Synchronization

13. Flush + FlushFlush + Flush
L1
Instruction
cache
Branch Predict.Isnt. Fetch
Pipeline(s)
Instruction decoder
Dispatch Integer
Cluster
2FPU
W.C. Cache
L1
Instruction
cache
L1
data
cache
Integer
Cluster
1
L1
data
cache
L2 Data Cache
shared
Core
Iface
Single Core
Shared L3 Cache (LLC)
Synchronization

14. Cache Side­Channel Attacks
➢ Both attacks require CPU cache flushing
➢ clflush or clflushopt
➢ filling the cache with already known data (as in
the Spectre attacks)

15. And than...And than...
MeltdownMeltdown
andand
SpectreSpectre
camecame

16. More architecture...
Floating Point
L1 D-Cache D-TLB
Schedulers
Integer
μop queues
Decoder
Trace Cache
Rename/Alloc
μop ROMBTB
BTB and I-TLB
BusL2CacheandControl
Thread 1: floating point

17. More architecture...
Floating Point
L1 D-Cache D-TLB
Schedulers
Integer
μop queues
Decoder
Trace Cache
Rename/Alloc
μop ROMBTB
BTB and I-TLB
BusL2CacheandControl
Thread 1: integer Thread 2: floating point

18. Meltdown mitigationMeltdown mitigation
kernel space
user space
All memory
➢ Protection
➢ Kernel Page Table Isolation
(KPTI)

19. MeltdownMeltdown
kernel space
user space
All memory
➢ exploits usually use
➢ Flush + Reload
➢ Flush + Flush
➢ TSX
➢ for better results the dying
processes should be on the
same core

20. Meltdown mitigationMeltdown mitigation
➢ Protection
➢ Kernel Page Table Isolation
(KPTI)
kernel space
user space
All memory

21. Spectre ­ MitigationsSpectre ­ Mitigations
Variant 1Variant 1
➢ Retpoline

22. Spectre ­ MitigationsSpectre ­ Mitigations
➢ Indirect Branch Restricted Speculation (IBRS)
➢ Single Thread Indirect Branch Predictors
(STIBP)
➢ Indirect Branch Predictor Barrier (IBPB)
Variant 2Variant 2

23. ➢ Indirect Branch Restricted Speculation (IBRS)
➢ disable branch prediction
➢ Single Thread Indirect Branch Predictors
(STIBP)
➢ isolate the predictions to hyper-thread
➢ Indirect Branch Predictor Barrier (IBPB)
➢ prevent user space predictions to leak to
kernel space (on every context switch)
Spectre ­ MitigationsSpectre ­ Mitigations
Variant 2Variant 2

24. All of the proposed All of the proposed
mitigations cost mitigations cost
performance in performance in
different scalesdifferent scales

25. https://www.phoronix.com/scan.php?page=article&item=linux-kpti-pcid&num=3

26. What others did?What others did?
➢ Monitoring and analysisMonitoring and analysis
➢ Capsule 8 ­ wrote on Jan 5Capsule 8 ­ wrote on Jan 5
➢ EndGame ­ shared their EndGame ­ shared their
research on Jan 08research on Jan 08
➢ I found out about EndGame and I found out about EndGame and
Capsule 8 on Jan 11, after we Capsule 8 on Jan 11, after we
had already started on our workhad already started on our work

27. Capsule 8 approachCapsule 8 approach
Kernel tracepoints and monitor Kernel tracepoints and monitor
for:for:
exceptions/page_fault_userexceptions/page_fault_user
Kernel perf countersKernel perf counters
– PERF_COUNT_HW_CACHE_OP_READPERF_COUNT_HW_CACHE_OP_READ
– PERF_COUNT_HW_CACHE_RESULT_ACCESSPERF_COUNT_HW_CACHE_RESULT_ACCESS
– PERF_COUNT_HW_CACHE_RESULT_MISSPERF_COUNT_HW_CACHE_RESULT_MISS
https://github.com/capsule8/capsule8/tree/master/examples

28. Capsule 8 approachCapsule 8 approach
This is nice, but not enough... This is nice, but not enough...
sincesince
Flush+Reload can be replaced by Flush+Reload can be replaced by
Flush+Flush to achieve the same Flush+Flush to achieve the same
result without actual page missresult without actual page miss
https://github.com/capsule8/capsule8/tree/master/examples

29. EndGameEndGame
They did not provide code They did not provide code
examples...examples...
However they explained a lot around However they explained a lot around
the statistics and using the CPU the statistics and using the CPU
performance counters.performance counters.
https://www.endgame.com/blog/technical-blog/detecting-spectre-and-meltdown-using-hardware
-performance-counters

30. Both examples by Capsule 8 and Both examples by Capsule 8 and
EndGame provide detection, but EndGame provide detection, but
little to no countermeasures.little to no countermeasures.

31. What is our setup...What is our setup...
First, our environment is shared First, our environment is shared
hosting and linux containers hosting and linux containers
hosting, where users may have root hosting, where users may have root
accessaccess

32. A decrease of CPU performanceA decrease of CPU performance
by 5­15% for one machineby 5­15% for one machine
is not much...is not much...
What is our setup...What is our setup...

33. A decrease of CPU performanceA decrease of CPU performance
by 5­15% for one machineby 5­15% for one machine
is not much...is not much...
but if you have hundreds or but if you have hundreds or
thousands of machines... thousands of machines...
the numbers become UGLY the numbers become UGLY 
What is our setup...What is our setup...

34. What we did?What we did?
Kernel moduleKernel module

35. What we did?What we did?
Kernel moduleKernel module
– detecting processes that had more then detecting processes that had more then
1 child dying with SIGSEGV1 child dying with SIGSEGV

36. Kernel moduleKernel module
– detecting processes that had more then detecting processes that had more then
1 child dying with SIGSEGV1 child dying with SIGSEGV
– when such process is detected it is when such process is detected it is
STOPPED, not KILLEDSTOPPED, not KILLED
What we did?What we did?

37. Kernel moduleKernel module
– detecting processes that had more then detecting processes that had more then
1 child dying with SIGSEGV1 child dying with SIGSEGV
– when such process is detected it is when such process is detected it is
STOPPED, not KILLEDSTOPPED, not KILLED
– only the root on the host machine can only the root on the host machine can
continue or kill this processcontinue or kill this process
What we did?What we did?

38. What we had to change?What we had to change?
– Introduced a per process counter of Introduced a per process counter of
its SIGSEGV childrenits SIGSEGV children
– check the counter on every reschedule check the counter on every reschedule
of the processof the process
– cpu pinning is allowed only to the cpu pinning is allowed only to the
host roothost root

39. Cache Line FlushCache Line Flush
– Limiting clflush and clflushopt Limiting clflush and clflushopt
effectively stops Flush+Reload and effectively stops Flush+Reload and
Flush+Flush attacksFlush+Flush attacks
– cache flush can be indirectly called cache flush can be indirectly called
when invalid instruction is issuedwhen invalid instruction is issued
– This greatly limits the options for This greatly limits the options for
executing Meltdown, by leaving only executing Meltdown, by leaving only
TSX instructionsTSX instructions

40. Cache Line FlushCache Line Flush
– both clflush and clflushopt are both clflush and clflushopt are
unprivileged instructions. Trapping unprivileged instructions. Trapping
them is not directly possiblethem is not directly possible
– We discussed different approaches:We discussed different approaches:
●
Inspecting the instructions of each Inspecting the instructions of each
binary, before it is executed and marking binary, before it is executed and marking
it cleanit clean
●
Inspecting the binary in parallel while Inspecting the binary in parallel while
the program is executingthe program is executing
●
Virtualizing the system and actually Virtualizing the system and actually
trapping the instructions after they have trapping the instructions after they have
been evaluated by the guest kernelbeen evaluated by the guest kernel

41. Cache Line FlushCache Line Flush
– adding noclflush on the kernel cmdline adding noclflush on the kernel cmdline
does NOT disable clflush!!!does NOT disable clflush!!!
– Events you can monitor for clflush:Events you can monitor for clflush:
●
L2_LINES_OUT.DEMAND_CLEANL2_LINES_OUT.DEMAND_CLEAN
●
MEM_LOAD_UOPS_RETIRED.L3_MISSMEM_LOAD_UOPS_RETIRED.L3_MISS
●
PERF_COUNT_HW_CACHE_LLPERF_COUNT_HW_CACHE_LL
●
PERF_COUNT_HW_CACHE_OP_READPERF_COUNT_HW_CACHE_OP_READ
●
PERF_COUNT_HW_CACHE_RESULT_ACCESSPERF_COUNT_HW_CACHE_RESULT_ACCESS
●
PERF_COUNT_HW_CACHE_RESULT_MISSPERF_COUNT_HW_CACHE_RESULT_MISS

42. TSXTSX
Transactional Synchronization Transactional Synchronization
eXtensions (TSX)eXtensions (TSX)
– Because of issues with the Because of issues with the
implementation, TSX instructions implementation, TSX instructions
should be disabled on Haswell CPUsshould be disabled on Haswell CPUs
– However if the microcode is not However if the microcode is not
applied your Haswell CPUs support applied your Haswell CPUs support
TSX :)TSX :)
– TSX instructions are supported on TSX instructions are supported on
Skylake...Skylake...

43. TSXTSX
– One thing that EndGame showed us is One thing that EndGame showed us is
that TSX instructions can be countedthat TSX instructions can be counted
– RTM_RETIRED.ABORTEDRTM_RETIRED.ABORTED

44. TSXTSX
– I considered lying to userspace by I considered lying to userspace by
reporting that TSX is not supported by reporting that TSX is not supported by
the CPU...the CPU...
●
but the cpuid instruction is unprivileged but the cpuid instruction is unprivileged
so trapping it is a non­trivial job so trapping it is a non­trivial job 

45. TSXTSX
– So relying on CPU counters is So relying on CPU counters is
currently the only sensible way of currently the only sensible way of
detecting if TSX is being used for detecting if TSX is being used for
Meltdown exploitsMeltdown exploits
– There is almost no code that has a There is almost no code that has a
legitimate use of TSX instructions, legitimate use of TSX instructions,
and on shared hosting we will most and on shared hosting we will most
likely not see such softwarelikely not see such software

46. TSX eventsTSX events
– RTM_RETIRED.STARTRTM_RETIRED.START ­ ­ Number of times we Number of times we
entered an RTM region. Does not count nested entered an RTM region. Does not count nested
transactionstransactions
– RTM_RETIRED.ABORTEDRTM_RETIRED.ABORTED ­ ­ Number of times an Number of times an
RTM execution aborted due to any reasons RTM execution aborted due to any reasons
(multiple categories may count as one)(multiple categories may count as one)
– RTM_RETIRED.ABORTED_TIMER ­ Number of RTM_RETIRED.ABORTED_TIMER ­ Number of
times an RTM execution aborted due to times an RTM execution aborted due to
uncommon conditionsuncommon conditions

47. you can find the list of Processor Monitor Unit
(PMU) events by running:
# perf list
Perf can be build from the linux kernel source
tree in tools/perf:
# make
# mv perf /usr/bin
TSX eventsTSX eventsTSX eventsTSX eventsTSX eventsTSX events

48. GlossaryGlossary
ALU ­ Arithmetic Logic Unit
AGU ­ Address Generation Unit
TLB ­ Translation Lookaside Buffer
BTP ­ Branch Target Predictor
BP ­ Branch Predictor
BTB ­ Branch Target Buffer
LLC ­ Last Level Cache
WB Cache ­ Writeback cache
W.C. Cache ­ Write combining cache
Trace Cache ­ execution trace cache

49. LinksLinks
Flush + Reload paper
Flush + Flush paper
EndGame research
Capsule 8 meltdown detection
Spectre & Meltdown attacks
Meltdown PoC
Collection of Speculation bugs info

50. Marian MarinovMarian Marinov
mm@siteground.commm@siteground.com