This paper describes log management in virtualized environments-its challenges and opportunities. We will cover the similarities and differences in logging for virtualized environments versus physical environments.

1. Where Logs Hide: Logs in Virtualized Environments
By Dr. Anton Chuvakin
WRITTEN: 2008
DISCLAIMER:
Security is a rapidly changing field of human endeavor. Threats we face literally change
every day; moreover, many security professionals consider the rate of change to be
accelerating. On top of that, to be able to stay in touch with such ever-changing reality,
one has to evolve with the space as well. Thus, even though I hope that this document will
be useful for to my readers, please keep in mind that is was possibly written years ago.
Also, keep in mind that some of the URL might have gone 404, please Google around.
This paper describes log management in virtualized environments-its challenges and
opportunities. We will cover the similarities and differences in logging for virtualized
environments versus physical environments.
Introduction to Logging
A beaten maxim proclaims that “knowledge is power,” but where do we get our knowledge
about information technology (IT) components such as computers, networking gear,
application frameworks, SOA web infrastructure and the like? The richest sources of such
information that is always available but often overlooked are the logs and audit trails that
are produced by these systems and applications. Through logs, audit trails and various
alerts, information systems often give signs that something is amiss, or an event logged in
the log files provides insight into future problems. Logs can also reveal larger weaknesses
–that may affect regulatory compliance and even IT governance, and, by extension,
corporate governance. However, more often than not, it’s difficult to extract information
from log files and distil the data into useful and usable or actionable information.
To start from the very high level, logs equal accountability. Wikipedia defines
accountability as " a concept in ethics with several meanings…often used synonymously
with such concepts as answerability, enforcement, responsibility, blameworthiness, liability
and other terms associated with the expectation of account-giving." There are many other
mechanisms for accountability in an organization, but logs are the most prevalent. And if
your IT staff is not accountable, neither is your business. Unless you take logs seriously,
you may be sending out the message that your organization shuns accountability. Along
the same lines, logs are also immensely valuable for meeting regulatory compliance. Many
recent US laws including HIPAA, GLBA, Sarbanes-Oxley (SOX) and others have
requirements related to log auditing and the handling of those logs (see my papers “Log
management in the age of compliance” and “Six Mistakes of Log Management”)
Let’s take a look at virtualization and what it means in terms of log collection and retention.
Introduction to Virtualization

2. Server virtualization makes it possible to combine multiple diverse systems onto a single
hardware platform, thus shrinking server, storage and networking costs, reducing power
requirements (through a direct decrease in consumed energy and cooling costs),
increasing utilization of existing computing resources and improving productivity. The
impact is significant; Garter reports savings of up to 25 percent due to server
consolidations and decreased hardware purchases.
Virtualization also simplifies server provisioning, increases the average workload per server
and shrinks server administration workloads, reducing the amount of required hardware
purchases. Organizations save money through better hardware utilization. Simplified
backup and recovery is also possible, because virtual machines can be brought back
online much faster than physical machines. Virtual platforms and their management tools
enable the smooth transition from a physical to a virtual environment.
It all sounds good, but what happens to logs, logging and log management when IT
environments are virtualized?
Logging Meets Virtualization
As one can guess, virtualization platforms present new sources of logs to manage. In
addition to having new log information to collect and analyze, we new challenges to logging
and log analysis arise, such as the potential need to review access logs collected while
virtual machine images were inactive. In addition, new opportunities for log management
are also present, such as ensuring new virtual images are pre-configured with central
logging capabilities. There may be ways to use logs to solve new problems, such as
monitoring health and uptime status of virtual platforms and application stacks. The
ubiquitous nature of log management allows the development of new operational, security
and compliance solutions for virtual infrastructures using the tools we already have.
What stays the same?
First, let’s review what stays the same. A virtual server is still a server – complete with
operating system and applications, and logs that must be collected, retained (for security
and compliance reasons) and analyzed, just as they do in “physical” environments. The
rest of IT infrastructure stays the same: Routers still route network traffic, switches perform
switching, firewalls and other network security devices perform their functions on network
traffic, etc. In other words, IT infrastructure with virtual platforms, hosts systems and guest
systems are largely the same as those with all physical elements; with all the usual logging
that needs to be managed. Similarly, networking between guest systems running on a
single virtual platform resembles networking between physical machines, and needs to be
monitored and audited just like on a physical network.
In a virtual environment, servers are still provisioned, modified and configured by system
administrators, and of course accessed and utilized by end users. Such activities create
audit trails that are collected and reviewed in just the same manner as are physical
environments. For example, if an MS SQL database server is running on Windows 2003
operating systems, but this Windows system itself sits atop of a Linux-based VMWare
host, both Windows logs and MS SQL audit trails must be collected and analyzed for
access violations, new user accounts, data access attempts or unauthorized changes to

3. database structures.
In short, the advent of virtualization is not a reason to throw away tools that work for you in
physical environments. They will continue to deliver value and help your IT and business to
operate efficiently, be secure and compliant with relevant regulations, especially given the
fact that the future belongs to a mix of physical and virtual environments.
What changes?
On the other hand, virtualization has brought a lot of new technologies (all with their own
logs) as well as new problems for IT departments to solve. Such problems might not have
any equivalent in the physical world, where “a server” always meant “a piece of hardware”
plus “an operating system” plus “one or more of user applications” running on it—a
worldview that virtualization is making obsolete.
A virtual platform comprises a hardware platform, operating system and a hypervisor, or
virtual machine software that enables other systems to run on top of it. Such a setup gives
way to several major changes:
1) New logs include hypervisor application logs, record virtualization-specific activity logs
(new guest image creation, guest operating systems startup, patch access, etc). These
logs must be understood by log management tools as well as the virtual machine
administrators.
2) Aggregation of servers on one hardware platform calls for stricter availability monitoring.
Indeed, recovering a virtual machine image from backups might be relatively simple, but
availability monitoring must still be stringent. Log management tools and possibly other
monitoring tools must be deployed with real-time alerting to notify the administrators of
impending fault and possible crashes or problems.
3) Stricter host platform security monitoring will help reduce the risk of breaches into the
virtual infrastructure world. Extensive logging, log collection and analysis will allow
thorough incident investigation. Such logs include security incident response and forensics
activity across virtual farms, as well as across massive SAN arrays that house virtual
machine images.
4) Management tools that enable organizations to deploy and control virtual server farms
introduce their own logs and logging challenges. For example, logging the activities of
server administrators means recording the provisioning, configuration and status changes
of virtual machines performed via such management tools.
5) As virtual machines proliferate across an enterprise’s IT infrastructure, physical hosts
are retired, an new technologies must be used to secure and manage the virtual machines.
Activity such as patching, management, configuration and deployment and migration of
virtual machines must be logged and monitored, just like in a physical environment.
Controlling and auditing these virtualization-specific activities makes another excellent use
case for logs.
Beware of Rogue Virtual Machines

4. Finally, “rogue” virtual machines pose a unique security problem. If users provision their
own virtual machines and their own guest systems, tracking such activities across the
organization, presents a worthy challenge – for example, if a unauthorized application, that
would otherwise be banned, runs in its own virtual image, enforcing the security policy
becomes harder since endpoint monitoring tools might not see through the virtualization
veil. Rogue machines deployed “in the cloud” via Amazon web services, for example,
present the ultimate challenge of this type. If a system resides on somebody else’s virtual
platform in the cloud, the chances of getting evidence of activities on such systems
becomes next to impossible.
Logging and Virtualization—The Good, the Bad and the Ugly
At this point it should be clear that changes that IT staff must face as virtualization
becomes a reality in the datacenter are indeed massive. For IT staff tasked with logging
activity across the infrastructure, these changes can be good, bad or ugly:
1) They’re good because it’s easier to provision systems with centralized logging already
enables. IT staff can also retrofit other systems by adding logging to the virtual image of
that system. Moreover, current logging tools such as LogLogic will still work – a major good
point.
2) They’re bad—or partly bad—because there are new logs to collect and analyze and new
activities to track and monitor. Virtual machines must be closely watched for availability and
security issues and to ensure they comply with policies and regulations.
4) They’re “ugly”—sometimes, because unmanaged virtual machines can pop up on the
organization’s systems or even in the cloud, violating IT policies and presenting significant
enforcement and investigation challenges.
Logs Help Virtualization
In addition to being affected by it, logging and log management can also augment
virtualization projects, especially in the areas of security, compliance and manageability..
Security: Logging creates a trail of accountability for users and, especially, those
privileged to access the underlying hypervisor. Tracking access to virtual machine hosts
system and inactive guest images creates a trail that can be used for monitoring and
auditing, as well as investigations for cybercrime or insider abuse. Perusing logs for
security-relevant failures, such as missing controls, unauthorized access or unapproved
changes is just as helpful in a virtual environment as it is in a physical environment.
Compliance: Recent mandates such as PCI DSS and others require logging, log collection
and retention, log analysis and review, and log protection. For example, logging is one of
the 12 PCI requirements (Requirement 10), whether the environment is physical or virtual.
Hence, logs from virtual machines must be given at least as much importance as logs from
physical environments
Manageability: Administrators and system operators benefit from logging, as well.
Monitoring for failures and errors as well as general virtual machine health is not possible

5. without effective log management.
Conclusion
Along with all the promise and benefits of a virtual infrastructure comes significant change,
requiring new ways for organizations to collect and manage logs. However, existing log
management tools such as LogLogic log management appliances can still be leveraged to
address these new logging challenges, and to optimize, secure and bring into compliance
newly virtualized IT infrastructures.
ABOUT THE AUTHOR:
This is an updated author bio, added to the paper at the time of reposting in 2009.
Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in the field of
log management and PCI DSS compliance. He is an author of books "Security Warrior"
and "PCI Compliance" and a contributor to "Know Your Enemy II", "Information Security
Management Handbook" and others. Anton has published dozens of papers on log
management, correlation, data analysis, PCI DSS, security management (see list
www.info-secure.org) . His blog http://www.securitywarrior.org is one of the most popular in
the industry.
In addition, Anton teaches classes and presents at many security conferences across the
world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia
and other countries. He works on emerging security standards and serves on the advisory
boards of several security start-ups.
Currently, Anton is developing his security consulting practice, focusing on logging and PCI
DSS compliance for security vendors and Fortune 500 organizations. Dr. Anton Chuvakin
was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked
at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the
importance of logging for security, compliance and operations. Before LogLogic, Anton
was employed by a security vendor in a strategic product management role. Anton earned
his Ph.D. degree from Stony Brook University.