The document discusses various techniques for anti-debugging including API based detection, process and thread block detection, hardware and register based detection, exception based detection, and modified code based detection. It provides examples of API based anti-debugging methods like FindWindow, IsDebuggerPresent, CheckRemoteDebuggerPresent, OutputDebugString, NtQueryInformationProcess, and NtSetInformationThread. The purpose of anti-debugging is to protect intellectual property by making reversing software more difficult through detection and hindering of debuggers.

1. Anti-DebuggingA Developers ViewpointMarch 13, 2008

2. Presenter BackgroundCurrentlySr. Security Researcher for Veracode, Inc.PreviouslySecurity Consultant - SymantecSecurity Consultant - @StakeIncident Response and Forensics Handler – US GovernmentWishes He WasInfinitely RichPersonal Trainer to hot Hollywood starletsGeneral all-around rockstar

3. What we will cover…Definition of Problem Statement and TermsWhy Bother?Isn’t this a futile effort?Standing On The Shoulders of GiantsBrief reference of anti-debuggingClasses of Anti-Debugging MethodsThe six styles of anti-debug fuSome Known Anti-Debugging MethodsNot enough time to cover them all

4. Problem Statement and Definition of Terms

5. Problem StatementAnti-debugging methods are consistently presented as machine code or assembly language constructs making knowledge transfer difficult and limiting widespread use of the techniques.

6. Let’s Do An ExperimentCan you tell what the following code does in five seconds or less?Be Honest!

7. Assembly Dumppush ebpmovebp, espsub esp, 0C0hpush ebxpush esipush edilea edi, [ebp+var_C0]movecx, 30hmoveax, 0CCCCCCCChrep stosdmovesi, espcall ds:__imp__IsDebuggerPresent@0cmpesi, espcall j___RTC_CheckEsptest eax, eaxjz short loc_411400movesi, esppush 0 ; uTypepush offset Caption push offset Text push 0 ; hWndcall ds:__imp__MessageBoxW@16 cmpesi, espcall j___RTC_CheckEspshort loc_41141Dloc_411400: ; CODE XREF: antidebug(void)+2Fjmovesi, esppush 0 ; uTypepush offset aNoDebuggerpush offset aNoDebuggerDetepush 0 ; hWndcall ds:__imp__MessageBoxW@16 cmpesi, espj___RTC_CheckEsploc_41141D: ; CODE XREF: antidebug(void)+4Ejpop edipop esipop ebxadd esp, 0C0hcmpebp, espcall j___RTC_CheckEspmovesp, ebppop ebpRetnantidebug@@YAXXZ endp

8. Assembly Dump

9. Assembly Dump – Highlight Important Bitsvar_C0 = byte ptr -0C0hpush ebpmovebp, espsub esp, 0C0hPush ebxPush esiPush edilea edi, [ebp+var_C0]movecx, 30hmoveax, 0CCCCCCCChRep stosdmovesi, espCall ds:__imp__IsDebuggerPresent@0 cmpesi, espCall j___RTC_CheckEspTest eax, eaxjz short loc_411400movesi, esppush 0 ; uTypepush offset Caption push offset Text push 0 ; hWndcall ds:__imp__MessageBoxW@16cmpesi, espcall j___RTC_CheckEspjmp short loc_41141Dloc_411400: movesi, esppush 0 ; uTypepush offset aNoDebuggerpush offset aNoDebuggerDetepush 0 ; hWndcall ds:__imp__MessageBoxW@16cmpesi, espcall j___RTC_CheckEsploc_41141D: pop edipop esipop ebxadd esp, 0C0hcmpebp, espcall j___RTC_CheckEspmovesp, ebppop ebpRetn?antidebug@@YAXXZ endp

10. Assembly Dump

11. How About In Cif (IsDebuggerPresent()){ MessageBox(NULL, L"Debugger Detected Via IsDebuggerPresent", L"Debugger Detected", MB_OK);} else{MessageBox(NULL, L“No Debugger Detected“, L“No Debugger ", MB_OK);}

12. Definition of TermsDebugger (Debugging)The act of detecting and removing bugs in an application.Anti-DebuggingAnti-debugging is the implementation of one or more techniques within computer code that hinders attempts at reverse engineering or debugging a target process.

13. Definition of TermsDumpingDumping of a running process in memory to disk so that the image can be executed as a separate binary. Anti-DumpingTechnique that is used to inhibit the dumping of a running process from memory to disk

14. Definition of TermsAnti-Anti-DebuggingTechniques used to detect and bypass anti-debugging effortsAnti-Anti-DumpingTechniques used to detect and bypass anti-dumping efforts.Anti-Anti-Anti-…. You get the picture

15. Why Bother?Isn’t this a futile effort

16. Why Should I Care?Protecting your intellectual propertyLaws can’t possibly work Neither do EULA or other soft methodsSlow down subversion of software registration CrackingImpede application feature and code theftLow costShort code segments equals quick implementationWrite a basic anti-debugging library for reuse

17. Why Should I Care?Raise the barImplement multiple layers of defenseMakes reversing an arduous taskIt’s no longer as simple as a debugger + disassembler + timeMalware analysisKnowledge of anti-debugging techniques can help your incident response and analysisDebug the random binary found on your accounting systemImplement IDS/IPS signatures in shorter timeframesExercise the brainKeeps the synapses firingImplementation (and bypass) is FUN!

18. Standing On the Shoulders Of GiantsA Brief Reference of Anti-Debugging

19. ReferencesBania, P, “Antidebugging for the (m)asses - protecting the env.”Brulez, N., “Crimeware Anti-Reverse Engineering Uncovered”“Lord Julus”, “Anti-Debugger and Anti-Emulator Lair”Liston, T., and Skoudis, E., “ Thwarting Virtual Machine Detection”Bania, P., “Playing with RTDSC”Quist, D., Smith, V., “Detecting the Presence of Virtual Machines Using the Local Data Table”Omella, A. A., “Methods for Virtual Machine Detection”Tariq, T. B., “Detecting Virtualization”Rutkowska, J., " Red Pill... or how to detect VMM using (almost) one CPU instruction”

20. ReferencesJackson, J., “An Anti-Reverse Engineering Guide”Falliere, N., “Windows Anti-Debug Reference”Gagnon, M. N., Taylor, S., and Ghosh, “Software Protection through Anti-Debugging”Ferrie, P., “Anti-Unpacker Tricks”OpenRCE: “Anti Reverse Engineering Techniques Database”Ferrie, P., “Attacks on More Virtual Machine Emulators”Brulez, N., “Anti-Reverse Engineering Uncovered”

21. Anti-Debugging Classes

22. Let’s Get Started

23. Let’s Get Started

24. Anti-Debugging Classes

25. Anti-Debugging ClassesAPI Based DetectionUsing operating system supplied API calls to determine if a debugger exists and/or is attached to our codeNo direct access of memory regionsRelies upon documented and undocumented operating system functionsProcess and Thread Block DetectionBypassing API calls and directly querying process and thread information to determine discrepancies that indicate the operation of a debuggerWhen direct API calls just aren’t enoughAvoids userland API hookingLower in the stack and closer to the OS results in more difficult bypass

26. Anti-Debugging ClassesHardware and Register Based DetectionDirectly accessing CPU register information specifically debug registers to check for hardware breakpointsNo longer relies upon software differences for detectionOne step lower in the stackException Based DetectionCreating an exception that is handled differently depending on the debugging state of a process

27. Anti-Debugging ClassesModified Code Based DetectionProcess understands what it *should* look like in memoryCompares the current picture against the excepted pictureFlags on differencesTiming DetectionTime calls used to determine execution deltas to be compared against a reasonable thresholdPrimarily used to detect single steppingCan also be used to detect breakpoints within blocks of code

28. API Anti-Debugging

29. Technology BackgroundX86 Systems and Microsoft Windows Operating SystemsWindows 2000/XP/VistaBreakpointsSoftware Breakpoints (Exceptions)Overwrites target location with INT03 (0xcc)Saves original opcodeBacks up one operation, replaces original opcode, restartsHardware Breakpoints (Faults)Debug registers: 4 for addresses, one for control, and one for statusEach of the first 4 can hold a memory address that causes a hardware fault when accessedDebug Registers (DR0 – DR7)Single SteppingTrap Flag in processor is setAny instruction will trigger a breakpoint when trap flag is set

30. API Anti-DebuggingProsGenerally easiest to implementEasiest to understandShort in lengthConsEasiest to understandRelatively easy to bypassMost frequently used

31. API Anti-DebuggingFindWindowHANDLE ollyHandle = NULL; ollyHandle = FindWindow(L"OLLYDBG", 0);if (ollyHandle == NULL) {MessageBox(NULL, L"OllyDbg Not Detected", L"Not Detected", MB_OK);} else {MessageBox(NULL, L"Ollydbg Detected Via OllyDbgFindWindow()", L"OllyDbg Detected", MB_OK);}Registry Key DetectionSOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug Debugger=exefile\\shell\\Open with Olly&Dbg\\commanddllfile\\shell\\Open with Olly&Dbg\\command

32. API Anti-DebuggingIsDebuggerPresent() Windows API if (IsDebuggerPresent()) { MessageBox(NULL, L"Debugger Detected Via IsDebuggerPresent", L"Debugger Detected", MB_OK);} CheckRemoteDebuggerPresent() Windows APICheckRemoteDebuggerPresent(GetCurrentProcess(), &pbIsPresent);if (pbIsPresent) {MessageBox(NULL, L"Debugger Detected Via CheckRemoteDebuggerPresent", L"Debugger Detected", MB_OK);}

33. API Anti-DebuggingOutputDebugString on Win2K and WinXPDWORD AnythingButTwo = 666;SetLastError(AnythingButTwo);OutputDebugString(L"foobar");if (GetLastError() == AnythingButTwo) {MessageBox(NULL, L"Debugger Detected Via OutputDebugString", L"Debugger Detected", MB_OK);} else {MessageBox(NULL, L"No Debugger Detected", L"No Debugger Detected", MB_OK);}

34. API Anti-DebuggingNtQueryInformationProcess() Detectionstatus = (_NtQueryInformationProcess) (-1, 7, &retVal, 4, NULL);printf("Status Code: %08X - DebugPort: %08X", status, retVal);if (retVal != 0) {MessageBox(NULL, L"Debugger Detected Via NtQueryInformationProcessProcessDebugPort", L"Debugger Detected", MB_OK);} else {MessageBox(NULL, L"No Debugger Detected", L"No Debugger Detected", MB_OK);}

35. API Anti-DebuggingNtSetInformationThreadlib = LoadLibrary(L"ntdll.dll");_NtSetInformationThread = GetProcAddress(lib, "NtSetInformationThread");(_NtSetInformationThread) (GetCurrentThread(), 0x11, 0, 0);

36. API Anti-DebuggingSelf Debugging with DebugActiveProcesspid = GetCurrentProcessId();_itow_s((int)pid, (wchar_t*)&pid_str, 8, 10);wcsncat_s((wchar_t*)&szCmdline, 64, (wchar_t*)pid_str, 4);success = CreateProcess(path, szCmdline, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi);… success = DebugActiveProcess(pid);if (success == 0) {printf("Error Code: %d\n", GetLastError());MessageBox(NULL, L"Debugger Detected - Unable to Attach", L"Debugger Detected", MB_OK);}if (success == 1) MessageBox(NULL, L"No Debugger Detected", L"No Debugger", MB_OK);

37. API Anti-DebuggingProcessDebugFlagshmod = LoadLibrary(L"ntdll.dll");_NtQueryInformationProcess = GetProcAddress(hmod, "NtQueryInformationProcess");status = (_NtQueryInformationProcess) (-1, 31, &debugFlag, 4, NULL);ProcessDebugObjectHandlestatus = (_NtQueryInformationProcess) (-1, 0x1e, &debugObject, 4, NULL);OllyDbg OutputDebugString() Format String VulnerabilityOutputDebugString(TEXT("%s%s%s%s%s%s%s%s%s%s%s"), TEXT("%s%s%s%s%s%s%s%s%s%s%s") );}__except (EXCEPTION_EXECUTE_HANDLER) {printf("Handled Exception\n");}

38. Exception Anti-Debugging

39. Technology BackgroundExceptionsVectored Exception HandlingLinked list of exceptions inserted before all other exceptionsStructured Exception HandlingLinked list of exceptions setup on a functional basisUnhandled Exception FilterA final catch all for exceptions to limit program crashesDebugger Interaction With ExceptionsFirst chance exceptions Before Vectored Exception HandlersModify and return execution to processIgnore and continuePass exception back to process for handling

40. Exception Based Anti-DebuggingProsSlightly more complexSometimes more difficult to bypass (debugger doesn’t know how to ignore the exception)ConsSlightly more complex Sometimes trivial to bypass (debugger ignore exception)Typically longer in length

41. Exception Anti-DebuggingINT 2D, INT 0x03, 0xF1 (ICE Breakpoint), 0xCC Detection__try { __asm {int 2dh; // substitute int 3 or __emit 0xF1, 0xCC for other triggers }}__except (EXCEPTION_EXECUTE_HANDLER) { flag = 1;MessageBox(NULL, L"No Debugger Detected", L"No Debugger", MB_OK);} if (flag != 1) MessageBox(NULL, L"Debugger Detected via int2d", L"Debugger Detected", MB_OK);

42. Exception Based Anti-Debuggingkernel32!CloseHandle __try {CloseHandle(0x12345678);}__except (EXCEPTION_EXECUTE_HANDLER) { flag = 1;MessageBox(NULL, L"Debugger Detected via kernel32!CloseHandle", L"Debugger Detected", MB_OK);}if (flag == 0) MessageBox(NULL, L"No Debugger Detected", L"No Debugger", MB_OK);

43. Exception Anti-DebuggingSingle Step Detection (Trap Flag)__try { __asm { PUSHFD; //Saves the flag registers OR BYTE PTR[ESP+1], 1; // Sets the Trap Flag in EFlags POPFD; //Restore the flag registers NOP; // NOP }}__except (EXCEPTION_EXECUTE_HANDLER) { flag = 1;MessageBox(NULL, L"No Debugger Detected", L"No Debugger", MB_OK);}if (flag == 0) MessageBox(NULL, L"Debugger Detected Via Trap Flag", L"Debugger Detected", MB_OK);

44. Exception Anti-DebuggingOllyDbg Memory Breakpoint DetectionmemRegion = VirtualAlloc(NULL, 0x10000, MEM_COMMIT, PAGE_READWRITE);RtlFillMemory(memRegion, 0x10, 0xC3); success = VirtualProtect(memRegion, 0x10, PAGE_EXECUTE_READ | PAGE_GUARD, &oldProt);myproc = (FARPROC) memRegion;success = 1;__try {myproc();}__except (EXCEPTION_EXECUTE_HANDLER) { success = 0;MessageBox(NULL, L"No Debugger Detected", L"No Debugger Detected", MB_OK);}

45. Exception Anti-DebuggingControl C Vectored Exception HandlingAddVectoredExceptionHandler(1, (PVECTORED_EXCEPTION_HANDLER)exhandler);SetConsoleCtrlHandler((PHANDLER_ROUTINE)sighandler, TRUE);success = GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0);Using the CMPXCHG8B with the LOCK PrefixSetUnhandledExceptionFilter((LPTOP_LEVEL_EXCEPTION_FILTER) error);__asm { __emit 0xf0; __emit 0xf0; __emit 0xc7; __emit 0xc8;}

46. Process and Thread BlockAnti-Debugging

47. Technology BackgroundProcess Environment Block (PEB)Process Base AddressHeap LocationMemory PointersGlobal FlagOS Version Information (Major/Minor)isDebugged Flag…Thread Information Block (TIB)Thread IDProcess IDPointer to process PEBError InformationException Information…

48. Process and Thread Block Anti-DebuggingProsDirect to the process and thread informationLess chance of detection via hookingHarder to bypass overallConsRequires use of undocumented structuresRequires understanding of runtime dynamic linkingMore complex to understand

49. Process and Thread Block Anti-DebuggingIsDebuggerPresent() Direct PEB Accessstatus = (_NtQueryInformationProcess) (hnd, ProcessBasicInformation, &pPIB, sizeof(PROCESS_BASIC_INFORMATION), &bytesWritten); if (status == 0 ) { if (pPIB.PebBaseAddress->BeingDebugged == 1) {MessageBox(NULL, L"Debugger Detected Using PEB!IsDebugged", L"Debugger Detected", MB_OK); } else {MessageBox(NULL, L"No Debugger Detected", L"No Debugger Detected", MB_OK);PEB ProcessHeap Flag Debugger Detectionbase = (char *)pPIB.PebBaseAddress;procHeap = base+0x18;procHeap = *procHeap;heapFlag = (char*) procHeap+0x10;last = (DWORD*) heapFlag;if (*heapFlag != 0x00) { … Found a debugger }

50. Process and Thread Block Anti-DebuggingNtGlobalFlag Debugger Detectionstatus = (_NtQueryInformationProcess) (hnd, ProcessBasicInformation, &pPIB, sizeof(PROCESS_BASIC_INFORMATION), &bytesWritten); value = (pPIB.PebBaseAddress);value = value+0x68; if (*value == 0x70) {MessageBox(NULL, L"Debugger Detected Using PEB!NTGlobalFlag", L"Debugger Detected", MB_OK);} else {MessageBox(NULL, L"No Debugger Detected", L"No Debugger Detected", MB_OK);}

51. Process and Thread Block Anti-DebuggingVista TEB system DLL pointerstrPtr = TIB+0xBFC; // Offset into the target structuredelta = (int)(*strPtr) - (int)strPtr; // Ensure that string directly follows pointerif (delta == 0x04) { if (wcscmp(*strPtr, hookStr)==0) { // Compare to our known bad stringMessageBox(NULL, L"Debugger Detected Via Vista TEB System DLL PTR", L"Debugger Detected", MB_OK); } else {MessageBox(NULL, L"No Debugger Detected", L"No Debugger", MB_OK); }} else {MessageBox(NULL, L"No Debugger Detected", L"No Debugger", MB_OK);}

52. Hardware and RegisterAnti-Debugging

53. Hardware and Register Anti-DebuggingProsDirectly checks the hardwareHarder yet to bypassFairly easy to implementConsNot a whole lot of research in this areaLimited number of techniques available

54. Hardware Register Anti-DebuggingHardware Breakpoint Detectionhnd = GetCurrentThread();status = GetThreadContext(hnd, &ctx); if ((ctx.Dr0 != 0x00) || (ctx.Dr1 != 0x00) || (ctx.Dr2 != 0x00) || (ctx.Dr3 != 0x00) || (ctx.Dr6 != 0x00) || (ctx.Dr7 != 0x00)){MessageBox(NULL, L"Debugger Detected Via DRx Modification/Hardware Breakpoint", L"Debugger Detected", MB_OK);} else {MessageBox(NULL, L"No Debugger Detected", L"No Debugger", MB_OK);}

55. Hardware Register Anti-DebuggingVmwaresldt detection__asm {sldtldt_info;}if ((ldt_info[0] != 0x00) && (ldt_info[1] != 0x00)) ldt_flag = 1;

56. Timing Based Anti-Debugging

57. Timing Based Anti-DebuggingProsSuper easy to implementConceptually easy to understandCan monitor blocks or individual instructionsConsVery easy to bypassCan be spotted a mile away

58. Timing Based Anti-DebuggingRDTSC Instruction Debugger Latency Detectioni = __rdtsc();j = __rdtsc();if (j-i < 0xff) {MessageBox(NULL, L“No Debugger Detected Via RDTSC", L“No Debugger Detected", MB_OK);} else {MessageBox(NULL, L"Debugger Detected Via RDTSC", L"Debugger Detected", MB_OK);}NTQueryPerformanceCounterQueryPerformanceCounter(&li);QueryPerformanceCounter(&li2);if ((li2.QuadPart-li.QuadPart) > 0xFF) {GetTickCount TimingtimeGetTime Timing

59. Paper Linkhttp://www.veracode.com/images/pdf/whitepaper_antidebugging.pdf

60. Questions

61. ContactTyler ShieldsSr. Security ResearcherVeracode, Inc.tshields@veracode.com