This document outlines a course on advanced malware reverse engineering. It begins with an introduction to malware types like viruses, worms, trojans, and ransomware. It then covers basic analysis techniques like viewing malware behavior and signatures. More advanced topics include static analysis using disassembly and dynamic analysis using debuggers. The goals of analysis are understanding malware functionality and enabling incident response. Analysis requires both static and dynamic methods to fully comprehend advanced threats.

1. CE0612-
Advanced Malware Reverse Engineering
Dr. Wajdi Elhamzi
w.elhamzi@psau.edu.sa
2023-2024

2. Outline
• Introduction to malware
• Basic analysis
• Advanced static analysis
• Advanced dynamic analysis
• Malware functionality
• Anti-reverse-engineering

3. CH1: Introduction to malware

4. What Is
Malware?
• Malware is a code that performs
malicious actions; it can take the
form of an executable, script, code,
or any other software.
• Attackers use malware to steal
sensitive information, spy on the
infected system, or take control of
the system.
• It typically gets into your system
without your consent and can be
delivered via various communication
channels such as email, web, or USB
drives.

5. Malicious actions?
The following are some of the malicious actions performed by
malware:
• Disrupting computer operations
• Stealing sensitive information, including personal, business, and financial
data
• Unauthorized access to the victim's system
• Spying on the victims
• Sending spam emails
• Engaging in distributed-denial-of-service attacks (DDOS)
• Locking up the files on the computer and holding them for ransom

6. Malware types
• virus
• worm
• backdoor
• trojan
• Spyware
• keylogger
• Botnet
• Remote administration tool (RAT)
• Adware
• Rootkit
• Banking malware
• Ransomware
• Downloader or dropper

7. Virus
• The first kind of malware that is known to
self-replicate.
• It is also called a file infector.
• Viruses survive by infecting and inserting
themselves into other healthy files on the
system.
Worm
A worm is a malware or a malware
functionality that spreads and infects other
computers, either via the network or some
physical means like the USB.

8. Trojan
A malware that masquerades as clean software and
is installed on the victim’s machine with the user’s
full knowledge, but the user is not aware of its real
malicious intentions.
Examples: Software hacking FB, Email, …
• A backdoor is malware that allows an
unauthorized user into devices, applications,
and networks.
• Attackers can gain backdoor access using a
command-line interface or other text-based
commands.
Backdoor
RAT: Remote Access Trojan, Remote
Administrative tools

9. Ransomware
Ransomware is a type of malware that locks and encrypts
a victim's data, files, devices, or systems, rendering
them inaccessible and unusable until the attacker
receives a ransom payment.
The first iterations of ransomware used only encryption
to prevent victims from accessing their files and systems.
• A logic bomb is a set of instructions in a program
carrying a malicious payload that can attack an
operating system, program, or network.
• It only goes off after certain conditions are met.
Logic Time Bomb

10. Spyware
Spyware is malicious software that enters a user’s
computer, gathers data from the device and user, and
sends it to third parties without their consent. A
commonly accepted spyware definition is a strand of
malware designed to access and damage a device
without the user’s consent.
Data sent from the infected device to attacker
• Adware (or advertising software) is the term used for various
pop-up advertisements that show up on your computer or
mobile device.
• Adware has the potential to become malicious and harm
your device by slowing it down, hijacking your browser, and
installing viruses and/or spyware.
Data sent from attacker to infected device
Adware

11. Fileless
Fileless malware is a type of malicious activity that uses native, legitimate tools built into
a system to execute a cyber attack. Unlike traditional malware, fileless malware does not
require an attacker to install any code on a target's system, making it hard to detect.

12. Advanced persistent threat (APT)
An advanced persistent threat (APT) is a sophisticated, sustained cyberattack in which
an intruder establishes an undetected presence in a network in order to steal sensitive
data over a prolonged period of time.
State sponsored
group
Remains
undetected for an
extended period
Advanced Persistent Threat
• Bootkits
• Rootkits

13. APT: Rootkit vs Bootkit
Rootkits Bootkits= boot capability + rootkits
• Designed to hide the existence of
certain process from normal methods
of detection, to enable continued
access to computer
• Examples: Stuxnet, Flame,..
• Are rootkits, in which the first point of
control is during the boot process.
• This allows the malicious program to be
executed before the operating system
boot ➔ run before antivirus
• Malicious boot record cannot just be
deleted or moved without damaging
the computer

14. Mass v. Targeted Malware
• Mass malware
– Intended to infect as many machines as
possible
– Most common type
• Targeted malware
– Tailored to a specific target
– Very difficult to detect, prevent, and remove
– Requires advanced analysis
– Ex: Stuxnet

15. Incident Response
• Case history
– A medical clinic with 10 offices found
malware on one of their workstations
– Hired a consultant to clean & re-image that
machine
• All done—case closed?

16. Incident Response
• After malware is found, you need to know
– Did an attacker implant a rootkit or trojan on
your systems?
– Is the attacker really gone?
– What did the attacker steal or add?
– How did the attack get in
• Root-cause analysis

17. Malware Analysis
• Dissecting malware to understand
– How it works
– How to identify it
– How to defeat or eliminate it
• Acritical part of incident response

18. The Goals of Malware Analysis
• Information required to respond to a
network intrusion
– Exactly what happened
– Ensure you’ve located all infected machines and
files
– How to measure and contain the damage
– Find signatures for intrusion detection
systems

19. Signatures
• Host-based signatures
–Identify files or registry keys on a victim
computer that indicate an infection
–Focus on what the malware did to the system,
not the malware itself
•Different from antivirus signature
• Network signatures
–Detect malware by analyzing network traffic
–More effective when made using malware
analysis

20. Static Vs. Dynamic Analysis
• Static Analysis
– Examines malware without running it
– Tools: VirusTotal, strings, a disassembler like IDA
Pro
• Dynamic Analysis
– Run the malware and monitor its effect
– Use a virtual machine and take snapshots
– Tools: RegShot, Process Monitor, Process Hacker,
CaptureBAT
– RAM Analysis: Mandant Redline and Volatility

21. Basic Static Analysis Basic Dynamic Analysis
• View malware without looking at
instructions
• Provide information about its
functionality (net connexion, Reg,
Domain)
• Running the malware in safe and isolated
lab environment
• Observing its behavior on the system
• Remove the infection
• Produce effective signatures
• Straightforward and easy
• Straightforward and easy
• Can be used by the most people without
deep program knowledge
• Fails for advanced malware and can miss
important behavior
• It doesn’t reveal the full extent of the
malware capabilities
• Fails for advanced malware and can miss
important behavior
• It doesn’t reveal the full extent of the
malware capabilities
Basic Analysis

22. Advanced Static Analysis Advanced Dynamic Analysis
• Loading the executable into a
disassembler and looking at the program
instructions which executed by the CPU
• Reverse-engineering with a disassembler
• Run code in a debugger
• Examines internal state of a running
malicious executable
• The ability to modify its execution path
and state
• It tell you exactly what the program does
• Extract detailed information from the
malicious executable
• It requires knowledge of disassembly
code, constructs windows OS concept and
internal
• It takes much longer time than basic static
analysis
• Fails for advanced malware and can miss
important behavior
• It doesn’t reveal the full extent of the
malware capabilities
Advanced Analysis

23. General Rules for Malware Analysis
• Don’t Get Caught in Details
– You don’t need to understand 100% of the code
– Focus on key features
• Try Several Tools
– If one tool fails, try another
– Don’t get stuck on a hard issue, move along
• Malware authors are constantly raising the bar