This document is a presentation about a crackme called Hackito Ergo Sum. It discusses the various techniques used to protect the crackme, including a verification algorithm using RC4 encryption, instruction mutation, control flow graph obfuscation, encryption layers, direct native API calls, anti-debugging methods, and ways attackers could potentially break it such as bruteforcing the encryption key or reversing the encryption algorithm. The presentation concludes by thanking the audience and opening the floor for questions.

1. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Hackito Ergo Sum Crackme
´
Eloi Vanderb´ken
e
eloi.vanderbeken (at) ens-cachan.fr
09 April 2011
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

2. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

3. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
About me
Last year of master’s degree in cryptology and software
´
security at Ecole Normale Sup´rieure of Cachan
e
Reversing software protection for 6 years
Participate to several Capture The Flags in my spare time
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

4. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

5. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Some facts and numbers
First crackme
2 months
6000 lines of python
Use TTASM and BeaEngine
Randomly generated
Only 3 valid solutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

6. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

7. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Algorithm
Custom RC4 (initial table changed, starts with ”HESFTW”)
Serial is used as a 256-byte key
Generate the first 7 DWORDs
Combine them with 0xDEADBEEF to generate a DWORD
Use it as a constant to calculate the length of the last layers
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

8. Introduction
Verification algorithm
Obfuscation
Encryption layers Instruction mutation
Direct native API call Control Flow Graph (CFG) obfuscation
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

9. Introduction
Verification algorithm
Obfuscation
Encryption layers Instruction mutation
Direct native API call Control Flow Graph (CFG) obfuscation
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

10. Introduction
Verification algorithm
Obfuscation
Encryption layers Instruction mutation
Direct native API call Control Flow Graph (CFG) obfuscation
Anti-X
How to break it
Possible evolutions
Instruction mutation
Make a static analysis to get freely modifiable registers
Use them to mutate instructions
Main rule: if your code use an instruction, it has to be used
everywhere ⇒ no difference between the useful code and the
garbage one
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

11. Introduction
Verification algorithm
Obfuscation
Encryption layers Instruction mutation
Direct native API call Control Flow Graph (CFG) obfuscation
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

12. Introduction
Verification algorithm
Obfuscation
Encryption layers Instruction mutation
Direct native API call Control Flow Graph (CFG) obfuscation
Anti-X
How to break it
Possible evolutions
Control Flow Graph (CFG) obfuscation
Before instruction mutation:
Duplicate some piece of code
Add a fake or non determinist test to choose which block will
be executed
After:
Make a static analysis to know which flags are set/unset for
each instruction
Add conditional jump which are never/always taken according
to the flags
Shuffle blocks
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

13. Figure: (incomplete) CFG of a layer made by IDA

14. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

15. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Layers
How to make a layer the most annoying possible?
Hide the end address of the encryption layer
Hide the size of the encrypted code
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

16. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Layers (continued)
Use Linear Feedback Shift Register (LFSR) to encrypt n times
both end address and a dword counter
At execution, decrypt them one time at each loop execution
When the dword counter has been decrypted, jump to the
decrypted address
Use indirect jump to avoid breakpoint:
cmp counter, CST setz cl jmp [ecx +
addresses]
Bonus: LFSR having the biggest possible period, you can add
as many fake comparison as you want
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

17. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

18. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Direct native API call
Get syscall numbers at initialisation
Use sysenter (do not work on 64bit systems)
Make random invalid or non invasives (ZwYieldExecution)
syscalls to prevent conditional breakpoints on
KiFastSystemCallRet
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

19. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

20. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Classical anti-debuggers tricks
Well known anti-debuggers trick made difficult to bypass because
of direct use of sysenter:
NtQueryInformationProcess + ProcessDebugPort
NtSetInformationThread + ThreadHideFromDebugger
NtSetContextThread to delete hardware breakpoints
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

21. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Anti-instrumentation framework
Detect hook on KiUserExceptionDispatcher and
KiUserCallbackDispatcher
Detect stack reuse by instrumentation code: place a constant
in esp-4, execute a sysenter, check if esp-4 has been changed
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

22. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Anti data-tainting
After execution of each part of the algorithm, result is copied using
indirect dependance
AntiTaintMemcpy (BYTE* source, BYTE *dest, int size)
{
for (int t =0 t < size ++t) {
BYTE val = 0
do {
BOOL isDiff = FALSE
if (source[t] != val)
isDiff = True
if (! isDiff)
dest[t] = val
++val
} while (val != 0)
}
}
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

23. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

24. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
So how to solve it?
API init Set memory breakpoint on ntdll PE
Anti-Dbg Don’t use debugger, inject a DLL and use
VectoredExceptionHandler to set HBP
Anti-HBP Replace ZwSetThreadContext syscall number by an
harmless or invalid one
ReadFile Use kernel debugger or replace syscall numbers by a
GDI one which will call KiUserCallbackDispatcher
Obfuscation Step the code :)
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

25. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
How to find a valid serial
Bruteforce (JB B´drune)
e
Use of the RC4 internal table created with the key published
(Kyriakos Economou and Mark Wodrich)
Generate a valid keystream, use a backward algorithm to find
an internal table which generate this keystream, find a key
which generate this table (Me :) )
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

26. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Plan
1 Introduction
2 Verification algorithm
3 Obfuscation
Instruction mutation
Control Flow Graph (CFG) obfuscation
4 Encryption layers
5 Direct native API call
6 Anti-X
7 How to break it
8 Possible evolutions
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

27. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Some ideas for the next year ;)
Recode all the crackme generation in C
Use XED (by intel) to disassemble and reassemble instruction
Multi-thread
Virtualisation
More mutation rules, more complex instructions (FPU, MMX,
SSE etc.)
More anti-X
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

28. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Thanks
Thank you !
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011

29. Introduction
Verification algorithm
Obfuscation
Encryption layers
Direct native API call
Anti-X
How to break it
Possible evolutions
Questions ?
Questions ?
´
Eloi Vanderb´ken
e Hackito Ergo Sum 2011