This document discusses security controls for a Red Hat Enterprise Linux virtualization environment hosting Top Secret VMs. It describes the hardware and software system configuration, including the use of KVM virtualization, Identity Management, and Satellite for patching. It also covers security concepts like SELinux and cgroups used to isolate VMs and limit resources. Hardening scripts are used to configure systems according to standards and continuous monitoring is enabled through SCAP and Satellite.

1. Defense in Depth 2014 | Frank Caviggia1
Top Secret KVM, Lessons Learned
from an ICD 503 Deployment
Frank Caviggia
July 30, 2014
Defense in Depth 2014

2. Defense in Depth 2014 | Frank Caviggia2
Overview
System Configuration
- Hardware
- Software
Security Controls
- Security Concepts
- Government Standards
- Hardening Scripts (STIG FIX, SSG)
- DISA STIG Kickstart DVD
SELinux Concepts
- What is SELinux?
- DAC and MAC
- Polyinstantiation and Multitenancy
KVM Security Features
- sVirt – SELinux Labels on VMs
- Multiple Firewall Levels
- cgroups – Control Groups (Limiters)
More Information

3. Defense in Depth 2014 | Frank Caviggia3
System Configuration

4. Defense in Depth 2014 | Frank Caviggia4
“Commodity Hardware”
Rack Servers
- Dell R200s (IdM)
- Dell R710s (RHN)
Blade Server
- HP C7000 with ProCurve 6120 Switch
- 4x HP BL460C G7 Servers
- HP D2200sb Storage Blade
System Configuration: Hardware

5. Defense in Depth 2014 | Frank Caviggia5
Red Hat Software
- Red Hat Enterprise Linux 6.5 Server x86_64
- Authentication (IdM) [389-DS(LDAP)/Kerberos]
- Red Hat supported version of FreeIPA
- Kernel Virtual Machine (KVM)
- Red Hat Enterprise Virtualization (RHEV) 3.3
- RHEV-M (Management Console for KVM Hypervisors)
- Red Hat Network (RHN) Satellite 5.6
System Configuration: Software

6. Defense in Depth 2014 | Frank Caviggia6
Kernel Cryptographic API "2.0" #1901 Certified, Level 1
Disk Volume Cryptographic API "2.0" #1933 Certified, Level 1
libgcrypt "2.0" #1757 Certified, Level 1
OpenSSH Client "2.0" #1791 Certified, Level 1
OpenSSH Server "2.0" #1792 Certified, Level 1
OpenSSL "2.0" #1758 Certified, Level 1
Openswan "2.0" #1859 Certified, Level 1
NSS (Freebl) 3.12.9.1 #1710 Certified, Level 1
NSS 3.12.9.1 #1837 Certified, Level 1
Red Hat Enterprise Linux 6 EAL4+
OSPP, including Labeled Security,
Advanced Audit, Advanced
Management, and Virtualization
Extended Modules
Dell
HP
IBM
SGI (report, target)
Red Hat Enterprise Linux 6:
- RHEL is Common Criteria Evaluated (Certified EAL 4+)
- FIPS 140-2 (Level 1 – Certified, Level 2 – In Evaluation)
- Linux Unified Key Setup (LUKS) Encryption (Data at Rest) – 8 Keys
- SSH uses AES256 (Counter mode) Encryption (Data in Motion)
- Web portals (Apache HTTPD) will require PKI authentication to view login page
System Configuration: RHEL, RHEV, KVM

7. Defense in Depth 2014 | Frank Caviggia7
KVM SPECvirt 2010 Results – KVM vs VMware
Kernel Virtual Machine (KVM):
- Type 1 Hypervisor (Bare-Metal) – runs as Kernel Module
- Utilizes native virtualization in Intel (VT-i,VT-d) and AMD hardware
- RHEL/KVM is Common Criteria Evaluated (Certified EAL 4+)
- Inherits Security (SELinux, Auditing) and Performance
- VM Limits (Leading Performance – SPECvirt)
- 160 CPUs
- 4TB RAM*
* As of RHEL 6.5/ RHEV 3.4 and newer
System Configuration: RHEL, RHEV, KVM (continued)

8. Defense in Depth 2014 | Frank Caviggia8
Red Hat Enterprise Virtualization Manager (RHEV-M):
- Web-based portal access to Virtual Machines (VMs)
- Role Based Access Control (RBAC) to VMs
- SPICE protocol (plug-ins for IE or Firefox) or Standalone Client
- Support for up to 4 independent monitors per VM
- Akin to vCenter in VMware
System Configuration: RHEL, RHEV, KVM (continued)

9. Defense in Depth 2014 | Frank Caviggia9
Red Hat Enterprise Linux Lifecycle:
- 10 years of support (up to 13 years with just security fixes)
- Common Vulnerabilities and Exposures (CVE) are fixed through
Red Hat Security Advisory (RHSA) process:
- Ensures IAVAs are patched
- Ensures system stability and support (backporting)
- There are no licenses, only subscriptions (stable budgeting)
- Upgrades are included with subscription
Support Lifecycle of Red Hat Enterprise Linux 6
System Configuration: RHEL, RHEV, KVM (continued)

10. Defense in Depth 2014 | Frank Caviggia10
RHN Satellite: SCAP Compliance Reporting
Red Hat Network Satellite 5.6
- IAVA patching and validation (patch management)
- File provisioning to connected hosts (configuration management)
- SCAP compliance scans (continuous monitoring) more on this later…
System Configuration: RHN Satellite

11. Defense in Depth 2014 | Frank Caviggia11
Security Controls

12. Defense in Depth 2014 | Frank Caviggia12
Security is like an onion – the more layers you peel the more you cry…
Goal: Create a secure virtualization environment using a standard set of packaged
scripts, configurations, and policies to be deployed across systems.
Controls are implemented through the following mechanisms:
- Hardening Scripts, Kickstart Installation
- Discretionary Access Controls (DAC)
- SELinux Policies
- Mandatory Access Controls (MAC)
- Network Controls (TCP_WRAPPERS, iptables, ebtables)
- Process and Memory Controls (cgroups)
- Administrative Controls (physical, policies, etc.)
- Continuous Monitoring (SCAP, RHN Satellite)
Security Overview

13. Defense in Depth 2014 | Frank Caviggia13
There are multiple government standards and regulations – some of which overlap:
Cross Domain Controls
NSA
SNAC
DISA STIGS
Common Criteria EAL 4+ System Security Controls
CAPP 1
RBACPP 2
LSPP 3
1 Controlled Access Protection Profile (CAPP)
2 Role-Based Access Control Protection Profile (RBACPP)
3 Labeled Security Protection Profile (LSPP)
NIST 800-53
(USGCB)
FIPS 140-2
Security Overview: Government Regulation…

14. Defense in Depth 2014 | Frank Caviggia14
Hardening will be applied by shell scripts, configurations, and policies based upon
several government standards and open-source projects to standardize configuration:
Apply Security Best Practices to Base Operating System
1 SCAP Security Guide - https://fedorahosted.org/scap-security-guide/
2 USGB Content -http://usgcb.nist.gov/usgcb/rhel/download_rhel5.html
3 Aqueduct Project - https://fedorahosted.org/aqueduct/
SCAP Security Guide1
- RHEL 6 SCAP, Security Configuration
NIST 800-53
- United States Government Configuration Baseline (USGCB)2
DISA Unix STIGs
- Aqueduct Project3
- Tresys’ Certifiable Linux Integration Platform (CLIP)4
NSA Security Configuration Guide5
- USB blocking, configurations, and other lockdowns (Gnome)
4 Tresys CLIP Project - http://oss.tresys.com/projects/clip
5 NSA SNAC Guide -
http://www.nsa.gov/ia/mitigation_guidance/security_configuration_guides/operating_systems.shtml#linux2
Hardening Scripts

15. Defense in Depth 2014 | Frank Caviggia15
The hardening shell script serves several purposes in hardening the system:
- Distributes ‘baseline’ system configurations and policies for authentication,
auditing, accounts, and services
- Modular code in folders and separate scripts allows for adaptation to meet
changing system and security needs of customer code
- Verifies application of hardening with logging, hardening can be re-applied in case
of modification of baseline, fits in with continuous monitoring
apply.sh
gen1000.sh
gen2000.sh
…
gen9999.sh
gen1000.sh
gen2000.sh
…
gen9999.sh
gen1000.sh
gen2000.sh
…
gen9999.sh
gen1000.sh
gen2000.sh
…
gen9999.sh
CAT I
CAT II
CAT III
CAT IV
nist1000.sh
nist2000.sh
…
nist9999.sh
NIST 800-53
NSA SNAC
nsa1000.sh
nsa2000.sh
…
nsa9999.shHardening Script Function (apply.sh)
Hardening Script: Implementation

16. Defense in Depth 2014 | Frank Caviggia16
Hardening scripts were packaged in RPM for the following reasons:
- Integrity verification
- Integrated version control and configuration management
- Distribute scripts via RHN Satellite Server
Check out the open source project here:
https://github.com/redhatgov/stig-fix-el6/
# rpm –V stig-fix-1.0.el6
Verify the integrity of an RPM
Hardening Script: Packaging

17. Defense in Depth 2014 | Frank Caviggia17
SCAP is implemented on Red Hat Enterprise Linux by OpenSCAP (oscap) and the
SCAP Security Guide (SSG) developed with collaboration with the NSA, NIST, and
DISA.
RHN Satellite can run SCAP Scans against a defined security baseline to check for
configuration compliance on a schedule. This helps to maintain continuous monitoring:
# oscap xccdf eval --profile stig-rhel6-server-upstream --results results.xml --report
report.html --cpe /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-dictonary.xml
/usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml
OpenSCAP XCCDF System Compliance Check
# wget http://www.redhat.com/security/data/oval/com.redhat.rhsa-all.xml
# wget http://www.redhat.com/security/data/metrics/com.redhat.rhsa-all.xccdf.xml
# oscap xccdf eval --results results.xml --report report.html com.redhat.rhsa-all.xccdf.xml
OpenSCAP XCCDF Patch (CVE) Compliance Check
- XCCDF (eXtensible Configuration Checklist Description Format)
- Creates checklist for security configuration on a target system
- OVAL (Open Vulnerability and Assessment Language)
- Standardized security information content
- CPE (Common Platform Enumeration) Dictionary
- Names and Metadata for security Evaluation
- CCE (Common Configuration Enumeration)
- Identifies mappings between SCAP security checks and STIG/NIST 800-53 settings
SCAP Terms & Definitions
Security Configuration Automation Protocol (SCAP)

18. Defense in Depth 2014 | Frank Caviggia18
The hardening script RPM was combined with a customized Kickstart to produce a
standardized installation DVD to help meet security requirements right from installation.
Screenshots of DISA STIG Kickstart DVD
Security Configuration: DISA STIG Kickstart DVD
More Information: https://people.redhat.com/fcaviggi/stig-fix/

19. Defense in Depth 2014 | Frank Caviggia19
SELinux Concepts

20. Defense in Depth 2014 | Frank Caviggia20

21. Defense in Depth 2014 | Frank Caviggia21
SELinux Policy ExampleWatertight Compartments in Ship Design1
1 Picture Source: Wikipedia – Bulkhead (Partition)
Security-Enhanced Linux (SELinux) was a research project sponsored by the NSA to
provide Mandatory Access Controls (MAC) to the Linux kernel
SELinux mainlined into the Linux kernel in August 2003 (2.6.0-test3), it was first
enabled for general use in Red Hat Enterprise Linux 4
Kernel enforcement based on “security context” provided by policies rather than standard
permissions. Think of it like a chroot jail on steroids or “watertight” compartments in ship
design
SELinux: Overview

22. Defense in Depth 2014 | Frank Caviggia22
Traditional Unix Permissions
- User, Group, Others (ugo)
- Read, Write, Execute (rwx)
Access Control Lists (ACLs)
- POSIX1 compliant ACLs standard in Linux filesystems (ext3, ext4, XFS, etc.)
- Extends DAC controls to specific user(s) and group(s)
1 Portable Operating System Interface EXchange
Discretionary Access Controls (DAC)
Concepts: Discretionary Access Control (DAC)

23. Defense in Depth 2014 | Frank Caviggia23
SELinux has 3 defined policy modes - Targeted (Default), Strict, and MLS.
Security Context implemented through extended attributes (xattr) in filesystem and
enforced by the Linux Kernel according to SELinux Policy
Unix concept of “everything is a file” (devices, processes, files, directories, etc.) Thus,
everything is labeled with a Security Context
SELinux policy defines the “watertight” compartments – the SELinux policy control how
users, services, files, and binaries interact
Policy is generally developed with software vendor when possible. Otherwise,
developing policy can be achieved through testing and evaluation giving “least privilege”
to allow completion of a job function
Security Context in SELinux
Concepts: Mandatory Access Control (MAC)

24. Defense in Depth 2014 | Frank Caviggia24
Type Enforcement (TE) used by Targeted policy (Default)
in SELinux
- The Linux Kernel enforces transactions between
processes and objects via domain transitions
- Further control can be specified using different policy
SELinux Domain Transitions
Compromised Apache process cannot access /etc/shadow
Concepts: Type Enforcement (TE)

25. Defense in Depth 2014 | Frank Caviggia25
1 IBM Developer Works Article “Improve Security with Polyinstantiation”
2 https://securityblog.redhat.com/2014/04/09/new-red-hat-enterprise-linux-7-security-feature-privatetmp/
Polyinstantiation1 is the process used on MLS systems to ensure data being processed by
users at separate security levels do so in isolated spaces to use to prevent unauthorized
access to data.
Data written to these directories will be stored in an independent directory at the security
level that they were written, particularly important for shared temporary directories (/tmp,
/var/tmp, /dev/shm/)
User will not see the redirection to a secure folder, SELinux handles the transition
transparently. See the “Private Tmp” feature in RHEL 72
Multitenancy extends the concept of polyinstantiation with cgroups and Linux Containers
(LXC) to ensure that applications are securely separated from each other through Type
Enforcement (TE) and MCS (the c0.c1023 attributes of the security level)
Multitenancy in OpenShift
Concepts: Polyinstantiation and Multitenancy

26. Defense in Depth 2014 | Frank Caviggia26
Kernel Virtual Machine Security

27. Defense in Depth 2014 | Frank Caviggia27
Each VM has their own “container” via SELinux Type Enforcement (TE) and
Multi-Category Security (MCS) which uses random compartments to keep the
VMs separate…
KVM Security: sVirt (SELinux for KVM)

28. Defense in Depth 2014 | Frank Caviggia28
Compromised VM uses hypervisor exploit to compromise other VMs
VS.
Compromised VM containment with KVM and sVirt (SELinux Labels)
KVM Security: sVirt (SELinux for KVM) (continued)

29. Defense in Depth 2014 | Frank Caviggia29
KVM Security: Multiple Firewall Levels

30. Defense in Depth 2014 | Frank Caviggia30
The ebtables firewall that enables basic ethernet frame filtering on a Linux network bridge,
logging, MAC (network address) NAT, and brouting.
The firewalls (iptables and ebtables) will be used to complement each other.
What can ebtables do?
# ebtables –A FORWARD –i vnet+ –-among-src ! 54:52:00:5b:1a:cd=192.168.0.100 –j DROP
Prevent IP-MAC Address Spoofing from VMs
# ebtables –A OUTPUT –i vnet+ –p IPv6 –j DROP
Drop All Outbound IPv6 Packets
• Ethernet protocol filtering
• MAC address filtering
• Simple IP header filtering
• ARP header filtering
• 802.1Q VLAN filtering
• In/Out interface filtering (logical
and physical device).
• MAC address NAT
• Logging
• Frame counters
• Ability to add, delete and insert rules; flush chains; zero counters
• Brouter facility
• Ability to atomically load a complete table, containing the rules you made,
into the kernel
• Support for user defined chains
• Support for marking frames and matching marked frames
KVM Security: Multiple Firewall Levels (continued)

31. Defense in Depth 2014 | Frank Caviggia31
cgroups (control groups) is a Linux kernel feature to limit, account and isolate
resource usage (CPU, memory, disk I/O, etc.) of process groups.
• Resource limiting: groups can be set to not exceed a set memory limit —
this also includes file system cache.
• Prioritization: some groups may get a larger share of CPU or disk I/O
throughput.
• Accounting: to measure how much resources certain systems use (e.g.
billing purposes)
• Control: freezing groups or checkpointing and restarting.
Dynamic Changes in Workload and Priority
(e.g. Number Crunching Overnight, Web Servers during Work Hours)
KVM Security: Control Groups

32. Defense in Depth 2014 | Frank Caviggia32
KVM Security: Secured Development Litterbox!

33. Defense in Depth 2014 | Frank Caviggia33
Questions?

34. Defense in Depth 2014 | Frank Caviggia34
More Information
DISA STIG Kickstart DVD:
http://people.redhat.com/fcaviggi/stig-fix/
Hardening Scripts:
https://github.com/RedHatGov/stig-fix-el6
SCAP Security Guide:
https://fedorahosted.org/scap-security-guide/
VDSM Hooks for RHEVM:
http://www.ovirt.org/VDSM-Hooks_Catalogue
Classification Banner:
https://github.com/RedHatGov/classification-banner