1. THERE’S GOLD IN
THEM THAR PACKAGE
MANAGEMENT DATABASES!
Phil Hagen
phil@lewestech.com
@PhilHagen
plus.google.com/+PhilHagen
Image: flickr.com/matthigh

2. WHY ARE WE HERE?
Goals:
Describe the value of package management databases during
the course of a Linux system forensic examination
Provide information with immediate benefit!
2

3. WHO IS THIS GUY?
Forensic/infosec consultant
Former DoD/IC contractor, USAF
Comm Officer (USAFA CompSci)
Course lead, FOR572: Advanced
Network Forensics Analysis
Linux guy since Slackware needed
a stack of floppies and an
unsupported SCSI controller
meant bootstrapping a kernel
compilation
3

4. OPEN SOURCE: INFINITE POWER
(ITTY BITTY LIVING SPACE)
Downloading and compiling source code is an amazingly powerful
aspect of most open-source software
Not viable for the large scale of a production environment
Does not address dependencies
No common install/uninstall process
Fun for the lab or a hobbyist environment, but a headache in
an operational environment
4

5. PACKAGE MANAGEMENT
SOFTWARE TO THE RESCUE!
Dependencies, file manifests, install/uninstall/upgrade scripts
Generally makes software management less of a headache!
Many solutions in widespread use among various distributions:
opkg (fork of ipkg): Embedded devices like QNAP NAS, etc.
dpkg: Debian and Ubuntu
tgz: Slackware
RPM: RedHat, CentOS, Fedora,many more
(part of Linux Standard Base)
5

6. …STILL NOT END-ALL/BE-ALL
Most incorporate higher-layer software to handle automatic
inclusion of dependencies and other “meta” functions and avoid the
much-feared “dependency hell” or “RPM hell”
apt-get / aptitude
YUM
RedHat Network
Not focusing on this higher-layer software
Some useful artifacts available - bad guys
known to “yum install nmap”
6

7. SCOPE FOR THIS PRESENTATION
Just looking at RPM
Most concepts apply to other package management standards
Notably dpkg
All examples created and tested on CentOS 6.5
Ideally: Same OS as subject (RPM library versions, etc)
Technically: Same version of RPM, BDB, related libraries
May have some success with unmatched versions, but beware!!
Presentation notes (including all commands) published soon
http://stuffphilwrites.com
7

8. SOURCES FOR RPM EVIDENCE
Populates Berkeley database
Lives in /var/lib/rpm/
Includes metadata for every
RPM-controlled file
User/Group ownership
Mode (aka permissions)
MD5 (er… SHA256)
checksum
File size
Major/minor number (For
entries in /dev/)
Symbolic link string (aka
“target”) for symlinks
Modification time
/var/log/yum.log
May have useful timestamped
history of install/remove/
upgrade actions
/var/log/rpmpkgs*
8

9. PRACTICAL USAGE
1. Mount all partitions from subject filesystem under /mnt/subject/
$ mount | grep subject
/dev/mapper/vg_centos6vm-lv_root on /mnt/
subject type ext4 (ro,noload)
/dev/mapper/loop0p1 on /mnt/subject/boot type
ext4 (ro,noload)
2. Run RPM commands with “--root /mnt/subject” option
Don’t trust the rpm(1) binary from a suspect system!
Note: This performs a chroot(2) for all operations - needs root
user privileges for validation (not query) actions
9

10. USE CASE:
FILE ORIGIN
$ rpm -qf filename
Identifies what package owns the specified file
Useful to answer “where did this file come from?” or to identify a file
as package-less
$ rpm --root /mnt/subject -qf /usr/sbin/sshd
openssh-server-5.3p1-94.el6.x86_64
$ rpm --root /mnt/subject -qf /etc/mail.rc Remember
chroot!
mailx-12.4-7.el6.x86_64
$ rpm --root /mnt/subject -qf /etc/crypttab
file /etc/crypttab is not owned by any package
10

11. USE CASE:
PACKAGE VALIDATION (1)
$ rpm -V packagename
Verifies contents of specified package
Compares expected (database) to actual (filesystem)
Displays files that failed =1 check, noting which checks failed
SM5DLUGT (Size, mode, MD5 (or SHA256!), major/minor nos,
link target, user, group, mtime)
Shows “?” in output if user running command lacks
permission to check (e.g. read access to generate checksums)
11

12. USE CASE:
PACKAGE VALIDATION (2)
WARNING!!!
Packages can include “verification scripts” which will execute
when the “-V” option is used!
You’re not planning to run arbitrary, unknown code on your
forensic workstation/VM, are you?
The chroot action needs root - these will execute as a
child to the sudo process!
Use the “--noscripts option with “-V” to prevent this
12

13. USE CASE:
PACKAGE VALIDATION (3)
Some files are expected to change after installation: config files!
Still show changes after installation, but denoted with a “c”
character
Missing files also noted in output as such
$ sudo rpm --root /mnt/subject
-V openssh-server --noscripts
no output
$ sudo rpm --root /mnt/subject
-V sudo --noscripts
S.5....T. c /etc/sudoers
13

14. USE CASE:
PACKAGE VALIDATION (4)
$ sudo rpm
S.5....T.
.M.......
S.5....T.
....L....
....L....
....L....
....L....
..5....T.
--root /mnt/subject -Va --noscripts
c /etc/sudoers
/proc
c /etc/maven/maven2-depmap.xml
c /etc/pam.d/fingerprint-auth
c /etc/pam.d/password-auth
c /etc/pam.d/smartcard-auth
c /etc/pam.d/system-auth
c /usr/lib64/security/
classpath.security
$ rpm -root /mnt/subject —V postfix --noscripts
missing
c /etc/postfix/master.cf
14

15. COOL FEATURE ALERT!
Many packages are GPG-signed
Independently verify package
without using compromised or
untrusted system
However: RPM database contents
not signed: Trojaned RPM package
reports no anomalies!
How can we use this great
feature to our advantage?
15

16. VALIDATE FILESYSTEM
AGAINST A PACKAGE FILE (1)
The “-p” option runs validation checks between filesystem contents and
RPM package file contents
Signed package files can be GPG-verified!
Avoids an untrusted RPM database entirely
Prevents false negative validation from compromised RPM installations
Relatively simple process:
1. Download trusted binary RPM file
2. Validate RPM file using GPG
3. Validate filesystem contents against package contents
16

17. VALIDATE FILESYSTEM
AGAINST A PACKAGE FILE (2)
Consider a system with a compromised RPM database, or a trojaned
installation of the Apache web server software
$ rpm --root /mnt/subject -V httpd --noscripts
S.5....T c /etc/httpd/conf/httpd.conf
$ wget http://mirror.centos.org/centos-5/5.10/os/
x86_64/CentOS/httpd-2.2.3-82.el5.centos.x86_64.rpm
$ rpm -K httpd-2.2.3-82.el5.centos.x86_64.rpm
httpd-2.2.3-82.el5.centos.x86_64.rpm: (sha1) dsa sha1
md5 gpg OK
$ rpm --root /mnt/subject -Vp
httpd-2.2.3-82.el5.centos.x86_64.rpm --noscripts
S.5....T c /etc/httpd/conf/httpd.conf
S.5....T
/usr/sbin/httpd
17

18. REAL-WORLD USE CASES (1)
Find all non-config files owned by an RPM that fails a verification check
$ sudo rpm --root /mnt/subject -Va --noscripts |
grep -v c
...
S.5....T
/var/www/awstats/lang/awstats-tt-tr.txt
S.5....T
/var/www/awstats/lang/awstats-tt-tw.txt
S.5....T
/var/www/awstats/lang/awstats-tt-ua.txt
.......T
/var/www/awstats/lib/blacklist.txt
S.5....T
/var/www/awstats/lib/browsers.pm
S.5....T
/var/www/awstats/lib/browsers_phone.pm
...
18

19. REAL-WORLD USE CASES (2)
File only config files owned by an RPM, which fail checksum verification
$ sudo rpm --root /mnt/subject -Va --noscripts |
grep ^..5..... c
S.5....T c /etc/pam.d/sshd
S.5....T c /etc/ssh/sshd_config
S.5....T c /etc/openldap/slapd.conf
S.5....T c /etc/sysconfig/ldap
S.5..... c /etc/sysconfig/saslauthd
S.5..... c /etc/security/limits.conf
S.5....T c /etc/logrotate.conf
SM5....T c /etc/snmp/snmpd.conf
S.5....T c /etc/sysconfig/snmpd.options
...
19

20. REAL-WORLD USE CASES (3A)
Find all files not owned by an RPM (This is going to be slow!)
$ cat find_orphans.sh
for file in $( sudo find /mnt/subject/etc -type f ); do
file=$( echo $file | sed -e 's//mnt/subject//' )
rpm --root /mnt/subject -qf $file 21 |
grep 'package$|directory$' |
sed -E 's/^(error: )?file (.*)(: No such file or
directory| is not owned by any package)/2/'
done
$ ./find_orphans.sh
/etc/crypttab
/etc/sysconfig/network
/etc/sysconfig/keyboard
/etc/sysconfig/iptables
...
20

21. REAL-WORLD USE CASES (3B)
Find all files not owned by an RPM (This is going to be slow!)
$ ./find_orphans.sh
/etc/crypttab
/etc/sysconfig/network
/etc/sysconfig/keyboard
/etc/sysconfig/iptables
...
21

22. COOL PARTY TRICK (1)
!
Use the “--queryformat” option to output only relevant/useful fields
from the RPM database
Provides 150 different tags that can be output for package or for each file
in a package
Available tags vary by version - online documentation is terrible
Use “--querytags” for listing specific to your version of RPM
!
Consider “RPMDBtoTimeline”…
22

23. COOL PARTY TRICK (2)*
$ for pkg in $( rpm --root /mnt/subject -qa ) ; do
rpm --root /mnt/subject -q $pkg --queryformat
[%{FILEDIGESTS}|%{FILENAMES}|0|%{FILEMODES:perms}|
%{FILEUSERNAME}|%{FILEGROUPNAME}|%{FILESIZES}|0|
%{FILEMTIMES}|0|0n] | sed -e 's/^|/0|/'
-e 's/|0|d/|0|d/d/' -e ’s/|0|-/|0|r/r/'
done
...
4398551f...a13988cf|/usr/share/doc/gamin-0.1.10/
callbacks.gif|0|r/rrw-r—r—|root|root|4514|0|1183556209|0|0
0|/usr/lib64/libmenuw.so.5|0|lrwxrwxrwx|root|root|15|0|
1282146079|0|0
0|/usr/share/cracklib|0|d/drwxr-xr-x|root|root|4096|0|
1308983949|0|0
...
* “Cool” claim not valid at all parties. YMMV.
23

24. IN CONCLUSION
RPM is a pretty cool way to eliminate known files from a Linux
system examination
Know the shortcomings in the RPM package database so you can
mitigate them
With a little shell scripting,
you can develop useful tools to
quickly and consistently
minimize input data