Buy vs. Build vs. Outsource: What’s Your Best Log Management Strategy?


Published on

This paper will examine the following considerations for choosing a log management solution for your organization:
• Why do you need log management in the first place?
• Should you build, buy or outsource your log management solution?
• What are the considerations for deciding on the appropriate log management strategy for your business?
• Is it better to use a combined log management strategy?

Published in: Technology

Buy vs. Build vs. Outsource: What’s Your Best Log Management Strategy?

  1. 1. Buy vs. Build vs. Outsource: What’s the Best Log Management Strategy? Dr. Anton Chuvakin WRITTEN: 2007 DISCLAIMER: Security is a rapidly changing field of human endeavor. Threats we face literally change every day; moreover, many security professionals consider the rate of change to be accelerating. On top of that, to be able to stay in touch with such ever-changing reality, one has to evolve with the space as well. Thus, even though I hope that this document will be useful for to my readers, please keep in mind that is was possibly written years ago. Also, keep in mind that some of the URL might have gone 404, please Google around. Logs—you don’t have to love them, but you have to have them. Logs are essential for adequate threat protection and intrusion discovery, incident response, forensics and even litigation support. They are used to check and enforce internal policies and procedures, as well as to measure IT performance. And they’re invaluable to IT staff when troubleshooting network, system and application issues. But what’s the best way to collect, store, manage, analyze and report on your log data? In other words, what is the best way to handle the “logging monster”? When deciding on a log management solution, you have many choices. You can build a solution of your own in-house, possibly utilizing the open source components. You can outsource log collection and management to a log management service provider, such as MSSP or, in the near future, to a SaaS provider. Or, you can buy an appliance or software solution from a software or appliance vendor. In addition, a preferable option may be to combine two of these options, so that you can take advantage of the benefits of both and mitigate their risks. Still, all of these strategies have both advantages and risks. This paper will examine the following considerations for choosing a log management solution for your organization: • Why do you need log management in the first place? • Should you build, buy or outsource your log management solution? • What are the considerations for deciding on the appropriate log management strategy for your business? • Is it better to use a combined log management strategy? Why collect logs in the first place? Let’s briefly review the nature, sources and importance of logs.
  2. 2. Logs come from everywhere within the IT infrastructure of an organization, whether large or small. Logs of relevance come from a wide variety of applications, network elements and endpoints and include audit logs, transactions, intrusions, connections and dropped connections, system performance records, user activities, and various alerts and other messages. More than 50 GB of logs can be generated daily by a large enterprise, resulting in nearly 20 terabytes of stored data in just a year. Why do you need to collect them? Logs are critical to ensuring and attesting to compliance and other business policies and regulatory mandates. With log data, you gain insight to records of user access — systems used, connection established, files viewed, emails sent — and you can identify successful and failed transactions, as well as system configuration changes in near real-time. Just as 20 years ago, logs are useful to system administrators, security analysts and IT managers. Logs can also help with troubleshooting network problems, and good log management can drastically simplify forensics activities and reduce e-discovery costs. A large percentage of log data is relevant to security; such logs include various audit records generated by the many devices and applications common in business environments. Even business applications generate security data – data that records access decisions or even indicates abuse or exploitation attempts. Collecting and analyzing all of this activity data across the IT environment (and even beyond IT, in the case of physical access monitoring) can illuminate malicious activity or unintentional security threats originating from within or outside the IT environment, so you can stop them faster. The Compliance Conundrum The importance of logs to compliance is increasingly clear to organizations of all sizes. Universally, industry regulations and governmental mandates require companies to collect, store and analyze logs— PCI DSS, SOX, FISMA, GLBA, HIPAA all include these requirements. There are really no exceptions. For example, NIST 800-53 (and NIST 800-92 to a larger extent) requires companies to capture audit records, regularly review them, automatically process them, protect audit info and retain logs. PCI requires companies to log and track user activities, automate and secure audit trail creation, review logs daily and retain an audit trail for at least a year. Furthermore, IT control frameworks like COBIT, ITIL and ISO 27002 also necessitate log collection, retention and analysis. COBIT, for example, recommends using logging to detect unusual or abnormal activities and determine root-cause analysis of mishaps. ISO guidance documents ask companies to maintain logs for information on changes, faults, corrections and capacity demands. Failure to comply with these requirements results in heavy consequences, ranging from monetary fines to essentially losing an ability to run your business to jail time. At the very least, companies can lose customers, reputation and revenue from the negative press associated with security breeches. Logs are no laughing matter; log management is no longer an option.
  3. 3. The Log Management Process A solid log management and intelligence solution is the only efficient way to create audit trails of network and system activity for all of the various uses of that data. Let’s take a look at what’s involved in the process. Log management tools solutions begin with log collection—gathering logs from critical systems, such as network devices, applications, databases and servers—and then storing them securely and unaltered in a centralized location for easy reporting and searching. By regularly reviewing logs, you can see failed logins, denied access attempts, unusual usage patterns – and get an overall feel of ongoing activity. Further, ongoing monitoring also calls for near real-time analysis and response in case action is needed. The ability to send alerts to key personnel when an event occurs is critical. Alerting allows us to monitor the logs and notify an operator if immediate action is needed. LMI allows you to create reports on collected log data, which is essential for compliance efforts. Both near real-time dashboard views and longer-term historical reports are needed. An efficient log management solution must allow organizations to store logs in their raw, unaltered form to ensure data integrity and forensic utility, and in a central repository for fast access. The ability to quickly search thorough large amounts of log data for investigative purposes is invaluable for incident response. Finally, LMI must allow for simplified yet secure log sharing. Typically, compliance and incident response are multi-team efforts that involve personnel from security to IT staff to management staff. Once the logs are collected and stored, fine-grained access control is needed to ensure that data is shared only with authorized stakeholders. Figure 1 illustrates key log management activities. Build, Buy or Outsource—Which Strategy Suits Your Business? Now that the drivers for log management as well as stages of a log management process are clear, let’s review how to actually do it! Deciding you need log management isn’t the hard part; deciding on how to implement it is. What’s the best strategy? Should you build your own solution, buy one off-the-shelf, or outsource log management as a service? Or, is there a combination of the three that would be the best bet? Let’s take a look at the pros and cons of each approach. Build it Many companies, especially smaller ones, choose to build their own LMI solutions. Indeed, you can try to build exactly the solution you need, with the platform, tools and methods you prefer, and aside from labor, there’s no up-front monetary cost. IT
  4. 4. professionals may even relish the challenges of creating a solution for the company and enjoy the challenges that are involved in “tackling the log beast.” But after a while, maintenance costs (due to an ever changing sea of log formats), log types, and log sources grow to overwhelming proportions - and the project often ends up killed. Since the solution is highly specialized, you will need highly specialized staff to add, change or repair the solution whenever necessary. Furthermore, these homegrown solutions are usually not scalable, so as the company grows and more data floods the network, changes, updates and ever-more-frequent overhauls are necessary—leading to even more labor and maintenance costs. During updates and ever-frequent overhauls, downtime can occur, costing you even more time and money. If you do decide to embark on a journey to “home-made log management,” there are a number of open-source tools that can perform some of the essential functions necessary for effective log management. Here are a few… • Log collection: Syslog-ng, kiwi, Snare, Project LASSO, and many others • Secure log centralization: stunnel/SSL, ssh or other encryption tools • Storage: MySQL or you can design your own – possibly indexed! - file-based storage • Analysis: SEC, OSSEC and OSSIM, Swatch, logwatch, logsentry and many other small scripts to solve one specific log-related problem Open source projects such as OSSEC and OSSIM also provide larger building blocks for your system by offering combined functionality. Over time, however, homegrown solutions are not practical, because the need to constantly update the support for changing log formats “gets them” in the end. According to Gartner researchers, “Although [home-grown log management] may prove effective for a limited set of data sources with clearly defined "strings" that the organization is searching for, most organizations quickly run into scalability issues, as well as issues using the data for situational awareness in support of incident response… In most cases, internally developed centralized application log solutions will fall short of meeting organizational requirements.” Outsource it Outsourcing log management is a low-cost way to get started with implementing LMI. Most likely, you won’t have to manage any equipment in-house and you won’t have to hire additional staff to run and maintain it. You’re basically paying someone else to worry about your problems. That sounds ideal, but there are some drawbacks, too. They’re still YOUR problems, and no one else is going to worry about them as much as you do, especially when regulatory compliance is at issue. You might find that a third party isn’t as careful about meeting your requirements in terms of IT policies and industry regulations. There is also a risk of SLAs slipping and potentially even losing control of your data. Plus, volume and log access challenges can arise when data collection and storage is outsourced to a service that may not be tuned into your
  5. 5. fluctuating business needs. To top it off, possible compliance violations will likely still fall on you and not on the service provider. Before choosing an outsourced solution, determine what portion of your logs will be managed by the service—is it all or just some? Know how you will gain access to your logs, so you can show them to auditors. Overall, for many organizations, especially the ones that are challenged to hire and retain IT staff and IT security professionals, the advantages of outsourcing are indeed compelling, and this option will continue to be viable and popular. Buy it As of today, procuring a log management tool from a vendor is fast becoming the most popular option. Fewer organizations are choosing to “build their own.” Vendor tools, such as LogLogic, have matured in recent years in both product capabilities and ease of deployment and operation. The option to buy a log management solution from the vendor is compelling: a commercial log management vendor will typically guarantee support for the log sources that you need, thereby mitigating the biggest risk and challenge of “home- grown” solutions’ constant updates, and will also expand support for new and changed logs and add new cutting-edge log analysis methods These tools can be very effective—you pay a set price and get a turn-key solution for log aggregation and analysis. All vendors offer support for wide ranges of diverse log sources, ongoing product improvements and innovations. Plus, if anything goes wrong, you have a scapegoat – a support person to scream at! But, as with other approaches, there are also risks. Sometimes skilled staff is needed to get value out of a purchased product, which still needs to be installed, run and maintained. Vendor longevity is also a problem—who do you turn to if the company who made the solution goes out of business? Choosing a company with experience will assure both vendor longevity as well as a stream of ongoing improvements. Combining approaches Because each strategy has its benefits and drawbacks, a combined strategy is often the best option. For example, you can purchase a solution and then enhance it with internal custom development on top of it. Or you can combine commercial vendor tools with open-source tools. You can also buy a tool and then outsource some of its management to an external provider. This allows you to maintain more control, but still lessen the workload on your IT staff. Combining solutions helps to mitigate some of the risks of individual solutions, however, it comes at a cost. Sometimes, you might even need to pay twice. Still, a larger upfront investment may prove cost-effective in the long run.
  6. 6. A “buy, then build on top” approach is often the most effective strategy to implementing a robust LMI solution that meets your specific – and evolving – business requirements. By combining the two, you can capture the advantages of both approaches, which include: • You get on-going support, upgrades and patches from the solution provider. • You’re assured reliable performance. • You can build the analysis tools you want. • You can present the data you want to the people who need it. • You can outsource the routine log management tasks to the vendor and only take on those you want to take on. In short, pick a vendor with a rich set of APIs that allows you to build on top of a commercial platform. Turn on the Logs! To conclude, if you do nothing else, turn logging on. Assess the role of log data in meeting compliance requirements, mitigating security risks, enabling audit and improving availability. Then implement the log management strategy that suits your business. Finally, avoid a build-only approach because it limits scalability and ends up costing more than it’s worth. If you have to build, build on top of a robust log management platform from a vendor. Considerations for Choosing an LMI Solution Before you decide on a log management approach and implement your new solution, you have a lot to consider. Trillions of log messages and hundreds of terabytes of data must be handled. Here are some questions you can ask yourself as you begin your quest for the best possible solution: 1. Are you collecting and aggregating 100% of all log data from all data sources on the network? 2. Are your logs transported and stored securely? 3. Are there packaged reports that suit your needs? Can you create the needed reports to organize collected log data quickly? 4. Can you set alerts on anything in the logs? 5. Are you looking at log data on a daily basis? Can you prove that you are? 6. Can you perform fast, targeted searches for specific data? 7. Can you contextualize log data (comparing application, network and database logs) when undertaking forensics and other operational tasks? 8. Can you readily prove that security, change management, and access control policies are in use and up to date?
  7. 7. 9. Can you securely share log data with other applications and users? ABOUT THE AUTHOR: This is an updated author bio, added to the paper at the time of reposting in 2009. Dr. Anton Chuvakin ( is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of books "Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy II", "Information Security Management Handbook" and others. Anton has published dozens of papers on log management, correlation, data analysis, PCI DSS, security management (see list . His blog is one of the most popular in the industry. In addition, Anton teaches classes and presents at many security conferences across the world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia and other countries. He works on emerging security standards and serves on the advisory boards of several security start-ups. Currently, Anton is developing his security consulting practice, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the importance of logging for security, compliance and operations. Before LogLogic, Anton was employed by a security vendor in a strategic product management role. Anton earned his Ph.D. degree from Stony Brook University.