This document discusses return address analysis for malware detection. It explains that return addresses provide important context about the execution flow and origin of API calls. Precisely tracking return addresses and API calls can help analyze application hijacking, detect unpacked/injected code, and identify abnormal system interactions that may indicate malware. While return address analysis provides useful insights, the document also notes limitations in fully detecting advanced exploits from external tools due to opportunities for a malware program to evade detection.

1. Amit Malik
SecurityXploded Research Group
Researcher @ FireEye Labs

2. The Content, Demonstration, Source Code and Programs presented here is "AS
IS" without any warranty or conditions of any kind. Also the
views/ideas/knowledge expressed here are solely mine and have nothing to do
with the company or the organization in which i am currently working.
However in no circumstances neither me nor SecurityXploded is responsible
for any damage or loss caused due to use or misuse of the information
presented here.
(c) SecurityXploded Research Group 2

3.  Objective
 Challenge
 Theory
 Properties of Call and RET
 Analysis
 Detection
 Conclusion
(c) SecurityXploded Research Group 3

4.  In defensive side of security we can define problems in two steps
 Analysis
 Detection
 Analysis – Understanding of the subject
 Detection – The Solution
 Detection depends on Analysis
(c) SecurityXploded Research Group 4

5.  At the top level the behavior of the malicious code can be divided into two
parts:
 Application execution hijacking
 System interaction and manipulation
 Application execution hijacking means deviation from the normal
execution path.
 System interaction and manipulation means using the system for malicious
activities.
(c) SecurityXploded Research Group 5

6.  Classic examples:
 Exploits
 Under normal circumstances (yes normal ) in execution hijacking the
execution will be transferred temporarily or permanently to stack or heap.
 After that the malicious code will always interact with the system (eg: file
system, Network, Processes etc.)
 *Normal hooks with code/DLL injection also comes under execution
hijacking but they are more part of system interaction so in execution
hijacking I am explicitly talking about the exploits.
(c) SecurityXploded Research Group 6

7.  Classic Examples:
 Malware binaries
 In this case the binaries or the code compromise the state of the
system.
 The scope here is the entire system so few things are difficult
(e.g.: detection).
(c) SecurityXploded Research Group 7

8.  The fundamental questions:
 Why a specific event/activity is happening in the system?
 Who started the event/activity.
 These questions in-fact are one of most difficult problems and at a
ground level even bigger than the software security problem.
 A very thin solution is try to identify the origin of the event/activity.
 In our case our focus is on API calls so if the API calls are the medium to
interact with the system then we need to identify the origin of the API
calls.
 It means that the return address is one of the most important pointer of
the entire execution cycle.
(c) SecurityXploded Research Group 8

9.  CALL ins:
 Push the address of the next instruction on to the stack i.e
Return Address
 Jump on to the destination address
 RET ins:
 Take the address from the current value of ESP (Return
Address) and load it into the EIP.
 The important point here is that the CALL and RET may be in a
relation or may not be in a relation but the return address will
always be on the stack [per ESP] so at any point of time if we hit
above ins then we can get the return address from stack [per
ESP].
(c) SecurityXploded Research Group 9

10.  Fundamentally Tracing is an approach to track the execution of the code.
 Roughly we can divide tracing in three parts:
1. Instruction level tracing i.e every instruction logging
2. Function and API level tracing i.e function and API call logging
3. API level tracing i.e only API call logging
 #2 and #3 are of our point of interest.
 But how we can log this information?
 Hooking
 Breakpoints
 Hooking is a clean approach but we need at least 5 bytes at the target
location on the other hand breakpoint is silly approach but we just need 1
byte.
(c) SecurityXploded Research Group 10

11.  Configuration-1: API calls Tracing
 Configuration-2: Function and API call tracing
 Configuration-1 is suitable for both analysis and detection.
 Configuration-2 is good for analysis.
Configuration-2 is also very useful for the analysis of normal applications.
(c) SecurityXploded Research Group 11

12.  Application execution hijacking:
 Configuration-1: API calls from heap or stack, basically calls from a
non-image mapped memory region indicates the hijacking of the
execution.
 Configurations-2: Function and API call tracing can provide in-depth
look into the execution cycle and possible locations for execution
deviation.
(c) SecurityXploded Research Group 12

13.  System Interaction and Manipulation
 Configuration-1:Based on the API calls and their return addresses
following information can be easily identified:
▪ Unpacked/run time generated/injected code in memory
▪ Interesting code segments
 My tool Malpimp is based on the same philosophy.
(c) SecurityXploded Research Group 13

14.  Benign File:
(c) SecurityXploded Research Group 14

15.  Malicious Files:
(c) SecurityXploded Research Group 15

16.  Execution Hijacking:
 Relatively easy* in comparison to malware binaries.
 In reality we have only two chances to detect this behavior from an external tool or
app.
 Bypass of DEP
 Execution of shellcode
 As mentioned in the theory we need to identify the origin of the API calls and if the
API calls are from a non-Image mapped memory region then we have some serious
problems.
 Practically speaking it is possible to make an exploit that is nearly impossible to
detect using an external tool because we have only two chances to catch the
abnormal execution.
 The checks for rop detection, heap spray, etc. are basically assumptions they are
not organic detection mechanisms. It all depends on the API call.
(c) SecurityXploded Research Group 16

17.  System interaction and Manipulation:
 The stages of unpacking/run time generated code can be easily
identified.
 Unpacking alone can not be really a detection mechanism.
 Abnormal behavior can be identified:
▪ Unpacking stages
▪ Some other events like (code injection etc.)
(c) SecurityXploded Research Group 17

18.  Return address is the invaluable pointer of the execution cycle.
 With the right implementation we can greatly improve the tasks related to
analysis and detection.
Thank You!
(c) SecurityXploded Research Group 18