Unix systems investigation and details

1. INVESTIGATING UNIX SYSTEMS

2. AN OVERVIEW OF THE STEPS IN A UNIX INVESTIGATION
• Before you can investigate a Unix system, you’ll need to set upyour forensic workstation and/or boot the image.
• Once you are ready to begin investigating the Unix system, the following actions provide the most likely way to
identify relevant evidence:
▼ Review all pertinent logs
■ Perform keyword searches
■ Review relevant files
■ Identify unauthorized user accounts or groups
■ Identify rogue processes
■ Check for unauthorized access points
■ Analyze trust relationships
▲ Check for kernel module rootkits

3. REVIEWING PERTINENT LOGS
• Unix operating systems have a variety of log files that can yield important clues during incident response.
• Not only are system activities such as logons, startups, and shutdowns logged, but also events associated with
Unix network services.
• Most log files are located in a common directory, usually /var/log.
• Some logs are placed in nonintuitive locations, such as /etc.
Network Logging
• The single most useful logging capability in Unix is the syslog (system log) file.
• The activities of syslog are controlled through the syslog configuration file, usually /etc/syslog.conf.
• A syslog daemon, syslogd, runs on the system to log messages.
• Syslog also offers the ability to log messages remotely, across a network.
• The syslog configuration file controls which types of messages are sent to which logs.
• Each line in the configuration file contains three fields:
 The facility field denotes the subsystem that produced the log file.
 The priority field indicates the severity of the log
 The action field specifies how the log will be recorded.

4. Remote Syslog Server Logs
• The log files generated locally by the syslog daemon are text files that are usually world-readable but writable
only by root.
• This means that any attacker who has gained administrator-level access can easily modify the syslog log files—
removing selected entries, modifying selected entries, or adding misleading entries.
• These modifications are nearly impossible to detect.
• If you suspect that an attacker has gained root-level access on the system where the logs are stored, do not trust
the logs.
• The only way to tell for certain if an attacker modified the log files is to perform redundant logging to a secure,
remote syslog server.
• In the event that a system is hacked and the log files are manipulated, or if the attacker deleted the entire log file,
a pristine copy should exist on the remote syslog server.
• The remote syslog server should be a hardened (secure) host with minimal access, preferably only console or
secure shell (ssh), which also takes advantage of system logging.
• The server’s accounts and passwords should be unique, to prevent access based on the compromise of passwords
from other systems.

5. TCP Wrapper Logging
• In addition to all of the applications that take advantage of the system logging capability, another extremely
valuable program that uses syslog is TCP Wrappers.
• TCP Wrappers is a host-based access control for TCP and UDP services.
• Any connection attempts to “wrapped” services are logged via syslog.
• The log entry provides a lot of valuable information: the time and date of the attempted logon, the hostname
(victim), the service (sshd), the account (root), and the IP address of the system that attempted to log on.
• Example that shows how a successful connection to a service is recorded:
Apr 26 20:36:59 victim in.tftpd[524]: connect from 10.10.10.10
• This entry shows that the host 10.10.10.10 connected to victim’s TFTP server .
• The correlation of connections and file-access times can be one of the investigator’s most powerful techniques.

6. Other Network Logs
• In addition to syslog, Unix systems can maintain other network activity logs.
• These logs are primarily service-specific, such as the log files for web servers.
• This log entry provides the following information:
▼ The time and date that the transfer occurred
■ The number of seconds that the transfer took
■ The remote host
■ The number of bytes transferred
■ The name of the transferred file
■ The type of file transfer (b for binary)
■ A special action flag (_ indicates no special action)
■ The direction of transfer (o represents outgoing; i is incoming)
■ The access mode (r is for real, as opposed to anonymous or guest)
■ The username (chris)
■ The service name (ftp)
■ The authentication method (0 for none)
■ The user ID (* indicates none available)
▲ The status of the transfer (c for complete)

7. Host Logging
• Unix provides a variety of log files that track host operations.
• Some of the more useful logs record su command execution, logged-on users, logon attempts, and cron job
(scheduled program) execution.
su Command Logs
• The su command allows a user to switch to another user ID during a session.
• Attackers sometimes use this command to attempt to gain root access to a system.
• Unix records every attempt to execute the su command on the system.
• The log shows the time and date of the su attempt, whether the attempt was successful, the terminal device from
which the user attempted to execute su, and the user ID before and after the su attempt.
• On some flavors of Unix, a separate su log file is stored in one of the log directories; on other flavors, su attempts
are recorded in the messages or syslog file.

8. Logged-on User Logs
• The utmpor wtmpfile is used to store information about users currently logged on to the system.
• The log file is named differently and stores slightly different information, depending on the flavor of Unix.
• The basic information stored is the name of the user, the terminal used to log on, and the time of the logon.
• The file is stored in a binary data format, rather than as a text file.
• Binary logs often contain more information than what is displayed with the default commands.
Logon Attempt Logs
• Logon attempts, both failed and successful, are recorded by default on most Unix systems.
• Along with the logon attempts for network services such as FTP or ssh, console logons are also saved in one of
the log files, such as the messages file on Linux systems.
Cron Logs
• Cron is a feature in Unix that allows users to schedule programs for future execution, and it is often used for
attacks.
• All executed cron jobs are logged, usually in /var/cron/log or in the default logging directory, in a file called cron.

9. User Activity Logging
• Along with logons, other types of user activities are recorded in Unix logs.
• Process accounting logs and shell history files record the commands executed by users.
Process Accounting Logs
• process accounting is a feature of Unix whereby every command run by every user is logged.
• This type of logging is not enabled by default.
• If the acct or pacct log file does not exist on the system, you will not be able to use this feature.
• If either of these files exists, you can use the lastcomm or acctcom command to review the contents of the file.
Shell Histories
• Users with interactive access to Unix systems have an associated command shell, such as the Bourne (sh), Korn
(ksh), or Bourne-Again (bash) shell.
• These shells provide the capability to log all commands, along with their command-line options.
• Typically, the history file is stored as a hidden file in the user’s home directory.

10. What Can Happen
• An attacker just gained root access to your system. One of the first steps the attacker takes is to delete the
.bash_history file.
• Then he links the file to /dev/null, rendering it incapable of logging commands.
Where to Look for Evidence
• Whenever you investigate a Unix system suspected of being compromised, check for shell history files.
• If the history feature is enabled and the history file does not exist, there is a good chance that the hacker deleted
the history file.
• If the history file exists as a link to /dev/null,, that is another strong indication that the system has been
compromised.

11. PERFORMING KEYWORD SEARCHES
String Searches with grep
• The powerful, flexible grep command is a primary tool for string searches.
• To perform a string search within a file, use the grep command as follows:
[root@lucky]# grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash
• Notice that the line in the passwd file with the string root inside appears as output.
• The passwd file is a text file.
• Now, let’s try grep on a binary file:
[root@lucky]# grep PROMISC /sbin/ifconfig
Binary file /sbin/ifconfig matches
• This time, the string does not appear. Instead, you see a notification that a file of type binary has a matching
entry.
• Different versions of grep have different functionality. The GNU versions of grep included with Linux are much
more full-featured than those found on many other, older flavors of Unix.
• In order to achieve the same results on a Solaris system, you need to use other utilities, such as strings, to first
pull the ASCII strings out of the binary file.

12. File Searches with find
• Another useful command for string searches is find.
• You can use the find command to find any filename that matches a regular expression.
• Example of searching the entire file system for a file or directory named “…”:
[root@aplinux /]# find / -name "..." –print
/home/mugge/MDAc/temp/.../root/...
• The first forward slash (/) indicates that the find operation will search the entire file system.
• The -name option specifies that the attribute to be searched on is the name of the file.
• The backslash () preceding each dot (.) is necessary to escape the special meaning of the dot, because, by default,
this character is a wildcard for regular expressions.
• The find command is helpful for many searches. It can search a file system for files that match a wide variety of
characteristics, including modification or access time, owner of file, string inside a file, string in the name of the
file, and so on.

13. REVIEWING RELEVANT FILES
Incident Time and Time/Date Stamps
• In order to search for files and directories that were accessed, modified, or created around
• the time of a suspected incident, you must first know the time of the suspected incident.
• The timeframe may be very specific, such as when a network IDS discovered and logged the attack as it
happened.
• The Unix file system saves three different timestamps for each file or directory:
▼ The atime, or access time, is the last time that a file or directory was accessed. This includes even read
access (such as cat filename).
■ The mtime, or modification time, records the last time a file was modified.
▲ The ctime, is similar to the mtime, but it records the last time the inode value was changed. This value
can change with events such as changing permissions or ownership.

14. Special Files
• Certain types of files and directories seem to regularly turn up in incidents.
• These files and directories include SUID and SGID files, unusual and hidden files and directories, configuration
files, and the /tmpdirectory
SUID and SGID Files
• Unix contains features known as set userid (SUID) and set groupid (SGID), which are designed to allow
programs to operate with higher privileges than those of the user running the program.
• SUID and SGID root programs are the source of most privilege-escalation attacks on Unix systems, and they are
also a favorite backdoor for attackers.
• A SUID root copy of /bin/ksh (the Korn shell) on most Unix systems will provide root privileges to any user
• who executes it.
• This is also known by attackers as a rootshell.
• To an investigator, a suspicious SUID root program is cause for alarm. To find all of the SUID or SGID programs
on a system, execute the following find commands:
[root@victim]# find / -perm -004000 -type f –print
[root@victim]# find / -perm -006000 -type f –print

15. Unusual and Hidden Files and Directories
• Attackers often hide files and directories from the casual observer.
• Within Unix, any file or directory that starts with a dot (.) is hidden from casual view; it will not appear in an ls
command listing unless the -a option is used.
• Furthermore, attackers often name files and directories with seemingly innocuous names, such as rpc.auditd for a
sniffer or /tmp/.X11-R5 for a directory.
• Especially common for directories is a name of just three dots (…). All of these names are similar to the names of
existing files and directories, and their appearance in a directory listing or a process table listing would not
immediately raise the suspicion of an administrator.
• The first stepto discovering this type of obfuscation is knowing when to take a closer look, such as in the case of
directories with multiple dots.

16. Configuration Files
• Configuration files are a key location of evidence during many incidents. With all of the built-in functionality of
the Unix operating system, a knowledgeable attacker can easily modify applications to perform evil tasks.
• Frequent targets include files that control access to the victim system, such as the TCP Wrapper configuration
files /etc/hosts.allow and /etc/hosts.deny.
• Attackers may modify or delete these files to allow certain computers to connect to the victim system at will.
• The Internet daemon configuration file inetd.conf (located in the /etc directory) controls many of the Unix
system’s network services.
• Services such as telnet, FTP, and TFTP (and many more services) are started via this file.
• An attacker may add entries to this file so that the victim system listens on many ports, or an attacker may enable
a previously disabled service such TFTP.

17. Where to Look for Evidence
• The inetd.conf file uses a partner file named /etc/services to define which ports are associated with which service.
In this case, you search the /etc/services file for the port 5500 entry:
[root@lucky /root]# grep 55000 /etc/services
telnet2 55000/tcp
• You identify a service named telnet2, which is associated with TCP port 55000. You search inetd.conf for this
service:
[root@lucky /root]# grep telnet2 /etc/inetd.conf
telnet2 stream tcp nowait root /usr/sbin/tcpd in.telnetd

18. Startup Files
• The Unix operating system has several locations that are used to start services and applications.
• Examples include cron, rc startup files, and user startup files.
• The cron facility is used to schedule the future execution of programs.
• The directory /var/spool/cron or /usr/spool/cron stores cron jobs for various users.
• Files in this directory are named after user accounts, and any jobs stored in those files are executed with the
privileges of that user.
• For example, jobs in the /var/spool/cron/root file are executed with root privileges.
• Another location of startupfiles is the rc directory.
• Usually named /etc/rc.d or something similar, this directory contains a listing of programs that start when a Unix
system boots.
• Programs like sendmail and portmapper traditionally are controlled by these configuration files
Tmp Directory
• By default, the /tmpdirectory is the only world-writable file system on a Unix system.
• This makes it a popular hangout for attackers and a favorite storage site for nefarious tools.
• Also, many publicly available exploits use the /tmp directory to store temporary files during privilege-escalation
attacks, and sometimes they leave trace evidence.

19. IDENTIFYING UNAUTHORIZED USER ACCOUNTS OR GROUPS
User Account Investigation
• User information is stored in the /etc/passwd file. This is a text file that you can easily review through a variety of
mechanisms.
• Every user on a Unix system has an entry in the /etc/passwd file. A typical entry looks like this:
lester:x:512:516:Lester Pace:/home/lester:/bin/bash
• The entry consists of seven colon-delimited fields: the username (lester), the password (shadowed in this case),
the user ID (512), groupID (516), GECOS field (for comments; Lester Pace in this case), home directory, and
default login shell.
Group Account Investigation
• Groupaccounts use the groupID shown in the /etc/passwd file as well as the /etc/groups file. A typical /etc/group
file looks like this:
$ cat /etc/group
root::0:root,ashunn
bin::2:root,bin,daemon
sys::3:root,bin,sys,adm
adm::4:root,adm,daemon
uucp::5:root,uucp

20. IDENTIFYING ROGUE PROCESSES
• Identifying rogue processes is much easier when examining a live system.
• During the initial investigation, you should have recorded all listening ports and running processes.
What Can Happen
• During your initial investigation, you dutifully record listening ports and running processes. Upon further
examination, you notice an anomaly with FTP:
[root@victim]# netstat –anp
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 519/inetd
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 519/ftpd
• Both telnet and FTP should be run from inetd, yet a separate FTP daemon appears to be running.
Where to Look for Evidence
• You examine /etc/inetd.conf and find that the FTP service has been disabled (by placing the # as the first
character):
[root@victim]# grep ftpd /etc/inetd.conf
#ftp stream tcp nowait root /usr/sbin/tcpd in.ftpd -l –a

21. CHECKING FOR UNAUTHORIZED ACCESS POINTS
• Unix is a fully functional, robust operating system. Over the course of its long history, Unix has continually added
functionality, and network services are no exception.
• A default installation of Unix offers a dazzling array of network services, including the Network File System
(NFS), telnet, finger, rlogin, and many others.
• Some of the most common access points that we have seen intruders take advantage of include X Servers, FTP,
telnet, TFTP, DNS, sendmail, finger, SNMP, IMAP, POP, HTTP, and HTTPS.
• Network services could be vulnerable, allowing intruders access to your system, or network services could
already be trojaned by a successful intruder.
ANALYZING TRUST RELATIONSHIPS
• Trust relationships within Unix systems were once a primary mechanism of attack.
• Trust can be established between Unix systems with a variety of services, the most popular of which include
rlogin, rsh, the Network Information Service (NIS and NIS+), NFS, and ssh.
• Trust relationships can be convenient time-savers for system administrators and users.
• If machine A trusts machine B, then the user on machine B can access machine A with no additional credentials.
• If you are a system administrator with dozens of systems to maintain, using this feature can be very enticing.
• Trust relationships are usually configured through files such as /etc/hosts.equiv or any .rhosts file in a user’s
home directory. Trust relationships can be established with ssh through shared keys and through NFS shares.

22. DETECTING TROJAN LOADABLE KERNEL MODULES
• Loadable kernel modules (LKMs), or kernel extensions, are found on the various flavors of Linux, BSD, and
Solaris.
• They extend the capabilities of the base operating system kernel, typically to provide additional support within
the operating system for device and file system drivers.
• LKMs can be dynamically loaded by a user with root-level access and they run at the kernel level instead of at a
normal user-process level.
LKMs on Live Systems
• Detecting trojan LKMs on a live system can be complicated because these tools actually intercept system calls
(such as ps or directory listing) to provide false information.
• They are specifically designed to prevent detection with traditional response methods.
• An example would be an external port scan compared to a port scan performed directly on the local suspect
system.
• Malicious LKMs are specifically designed to intercept system calls and provide false information.
• This is a key difference between rootkits and LKMs.
• When a traditional rootkit is installed, commands such as ps and netstat are modified or replaced.

23. LKM Elements
• In some cases, the intruder uploads and compiles the source, and successfully installs the LKM; however, she
forgets to delete the actual LKM source files!
• When this happens, you may not only discover the presence of the LKM, you may also find additional
configuration information. Here is an excerpt from a discovered Adore make file:
CFLAGS+=DELITE_CMD=102993
CFLAGS+=DELITE_UID=30
CFLAGS+=DCURRNT_ADORE=42
CFLAGS+=DADORE_KEY=”batman”
• The make file not only contains the Elite command, Elite UID, and Adore version number (4.2), but it also has
the Adore key. The DADORE_KEY value contains batman, the intruder’s password!
• If the intruder didn’t rename or hide the files, the following command would find the startadore script, one of
several associated files, if it existed on the root file system.
[root@curtis ]# find / -name startadore –print
/tmp/…/startadore