MERGE 2013 THE PERFORCE CONFERENCE SAN FRANCISCO • APRIL 24−26White PaperSecurity of source code is an evolving domain becauseinternal networks̶once considered to be safer̶areno longer trusted environments. Since enterpriseperimeters are no longer sufficient defense boundaries,the focus shifts to using defense-in-depth strategies tosecure internal systems that hold product code.Source Code Protection:Evaluating Source Code SecurityStephanie Woiciechowski, GSECEMC Product Security Office, EMC Corporation
2 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.Introduction: Threats/ObjectiveAttackers have multiple motivations for focusing on source code such as crime, espionage,hacktivism, and terrorism. Attacks may result in theft of source code that damages acompany’s reputation, allows attackers to search for vulnerabilities or other methods of attackin your code, or undermines competitive advantages. From a different angle, the attacker mayattempt to alter your source code to undermine product integrity and create weaknesses inyour customer environments.Traditionally, we have relied on basic system security hygiene along with the security ofcorporate (or private) networks to protect source code. Today, attacks have evolved to obtainand quietly maintain access on the network so there is greater pressure to understand thelevel of defense in depth needed to protect product engineering systems, such as source coderepositories. In August 2011, the Security for Business Innovation Council told organizations:“Consider that no organization is impenetrable. Assume that your organization might alreadybe compromised and go from there” (Security for Business Innovation Council Initiative, 2011).Attackers are looking for long-term access to your network and data. They are willing to investthe time and effort to go slowly, build multiple points of access, and learn your network betterthan you do. In this environment of new threats, we need to expand security awarenessbeyond security experts so software configuration management (SCM) administrators anddevelopment teams can take a larger role in supporting security.As a result of this changing landscape, we must update how we secure source code. Thiswhite paper offers a security-focused overview for non-security professionals to create sourcecode environments with defense-in-depth solutions. It starts with an exploration of commonsecurity criteria, describes how to analyze your source code environment, and providesexamples of managing change to implement security solutions to better protect your sourcecode.Defining Security Criteria: Policies and StandardsIdentify the stakeholders with an individual interest in protecting the company’s source code:security practitioners, legal advisors, program managers, source code administrators, anddevelopers within your company. Reach out to them to explore different security perspectivesand use cases for your organization. The goal of these conversations is to identify weak pointsin your environment (vulnerabilities) to define how someone might attack your environment(threats) and the results of a successful attack (impacts) so you can develop a business casefor improving security in your environment. Consider questions such as these:• What could happen if our source code was modified?• What could happen if our source code was stolen?• How would we recover and how much would this cost?Use the business case to get buy-in from your executives, ensure that you are working towardclear goals, and get the correct people involved in making decisions.With an approved business case, begin constructing visible policies. Policies describe thegeneral goals without focusing on implementation details and frame the objective of the goalsto cover even unforeseen circumstances. Invite the stakeholders to form a task force oradvisory team to have meaningful conversations about what is necessary and practical and
3 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.how to address accountability.Your task force will also develop the standards that define the rules supporting your policies.Your policies establish what needs to be done to protect source code; standards describe howto do it in a tool-agnostic way. Whether you are using Perforce Software Version Managementor ClearCase should be irrelevant to the standard because the standard could be equallyapplied to both. It is essential to have cross-team collaboration to develop standards to ensureboth adequate coverage and feasibility. This approach provides relevant stakeholders with avoice in the process so you obtain security-related improvements and compliance in a mannerthat is achievable, measureable, and maintainable.Common Defense-in-Depth StrategiesDefense in depth is “the concept of protecting a computer network with a series of defensivemechanisms such that if one mechanism fails, another will already be in place to thwart anattack” (McGuiness, 2001). The techniques our security and IT teams have been using toprotect the network are not dead. They still provide a layer of defense but need to besupplemented. Defense-in-depth strategies will support your approach to developing astandard.Data ClassificationYou may be familiar with military data classifications: Classified, Secret, Top Secret. Dataclassification reflects both the value of the data and the risk to the data so you can select theappropriate security solutions. By defining and applying data classifications to your sourcecode, you can frame how to protect different types of source code. Source code intended to bereleased under an open source license may deserve different security rules than the sourcecode that makes up your core design. Classification may take into account regulatoryrequirements (e.g., cryptography) or customer requirements (e.g., FIPS compliance). Forinspiration, look at resources such as FIPS PUB 199 Standards for Security Categorization ofFederal Information and Information Systems (FIPS PUB 199, 2004) or the ISO 27000 seriesof information security standards.Identity/AuthenticationAuthentication means proving your identity in a way that no one can later say it wasn’t you(non-repudiation). Authentication is usually achieved by combining a username (to provide anidentity) with a secret (such as a password or one-time token) to be validated. In the softwaredevelopment context, you want to identify who broke the build, and authentication providesthat information; Bob Developer cannot dispute he broke the build when his commit was theonly one since the last successful build. In the security context, you want authentication tosupply non-repudiation that only authorized individuals are touching the code. Non-repudiationcan only be achieved through proper authentication.By adopting centralized authentication mechanisms, you protect source code with yourorganization’s password policies; accounts can be disabled quickly when someone leaves theorganization or changes roles. Solutions such as secure LDAP or Active Directory can supportaccess groups while two-factor authentication—which combines something you know withsomething you have to re-create your secret—can provide enhanced security.
4 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.Some people argue that having local accounts separate from the centrally managed accountsadds another “layer” of security, but local accounts risk complicating the deprovisioningprocess and correlating user activity throughout the environment. “Business process flaws,departmental silos, and a lack of automation all stand in the way of streamlining this so-calleddeprovisioning process. And when organizations don’t get a handle on the orphan accountsleft behind by an ineffective deprovisioning process, they leave themselves open to fraudulentaccount use and a lack of visibility” (Chickowski, 2013). Having inconsistent account IDsacross systems makes it hard to trace user activity across the environment. If I have bob.smithand barbara.smith in the corporate directory, who is bsmith in my local account? This becomeseven more important when we talk about supporting enterprise logging and monitoring later.Access ControlsCombining data classification with authentication provides the data necessary to establishaccess controls to govern who has access to data. The principle of least privilege means youonly have access to the bare minimum you need to do your job.With proper access controls in place, a malicious insider working on Project A is restricted fromProject B’s code, limiting the scope of exposure to attack. This is not just a matter of nottrusting employees. The developer on Project A may be an honest, dedicated employee buther credentials were compromised; now, access controls limit the data available to theattacker.The principle of least privilege can be difficult to implement. The idea of limiting accessappears to go against many usual development practices: How do we share common code?Can a project be agile if everyone can’t commit everywhere? However, even in the opensource world, we can find examples of access controls: The Apache Way describes howApache projects allow anyone to contribute, but only committers can actually write codechanges to the official repository (The Apache Software Foundation, 2012).IsolationProperly configured firewalls are still a vital component for isolating network access andproviding logical protection to the source code. Source code repositories typically have at leastone layer of isolation behind firewalls on the internal network. Hardware hosting repositoriesused to build the product also should be physically isolated. The hardware should be behindlocked doors with limited access reducing the risk of physical tampering.Recently, a trend toward distributed solutions, such as Git, or SaaS solutions, such as PerforceOnDemand, requires extending how we apply isolation. Isolation still plays a role, but thetechniques are evolving. In a distributed solution, one repository should be identified as the“gold” repository, which is the definitive source for all code contributing to a final build. Beforeselecting a SaaS solution, you need to understand how the SaaS provider supplies isolationfor your environment. The hardware for your “gold” repository or at your SaaS provider’s facilitystill must be protected from unauthorized physical access, and the firewalls should isolate yoursource code environment from that of other customers.For additional isolation protection of exceptionally sensitive source code, the entiredevelopment and build environment can be physically and logically isolated. You can employ avirtual desktop infrastructure (VDI) solution to provide limited access to source code in a highly
5 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.protected network environment where only the final product ever leaves the environment.Classifying the data first will help you determine if additional protections such as this arewarranted; these additional controls incur extra costs and require changes in the developmentprocess. Usability is key to successfully deploying an enhanced security environment becauseend users still need an environment in which they can work efficiently and effectively. Be sureto prototype the solution with actual users and get their buy-in.Data ProtectionWhile isolation addresses security of your source code where it lives on your servers, dataprotection considers security of your source code in other places.If we assume our network has been compromised or if we’re using a SaaS solution, we want tomake sure source code is protected in transit over the network; we don’t want source codeleaked or changes made mid-flight by an attacker. Many source code tools provide an SSLsolution to encrypt data between the client and server applications. Using these solutions—along with the accompanying PKI solution to mitigate man-in-the-middle attacks—supports amore secure coding environment.Backup solutions aren’t just an issue for business continuity; they are also another area ofpotential vulnerability. When we’ve locked up our source code repository hardware, the backuptapes may become easier targets for a thief. Backups can also be a target for people looking totamper with your code, if they can modify the data on the backup and force you to restore thatdata. When a backup is no longer needed, it should be securely disposed of throughtechniques such as secure overwriting or physical shredding.Similar to backups, hardware decommissioning presents another area of vulnerability. Whenyou replace a hard drive, the device must be securely disposed of to prevent access to yoursource code. If the hard drive is functional, consider securely clearing or sanitizing the driveusing the National Institute of Standards and Technology’s Guidelines for Media Sanitization(NIST SP800-88) (Kissel, Scholl, Skolochenko, & Li, 2006).Even current hardware should be evaluated. In larger organizations, repository storage is oftenmounted in SAN or NAS storage, which requires evaluating access to another physical deviceand the processes used to securely dispose of them.For additional protection, consider implementing a data loss prevention (DLP) tool. With a DLPsolution, you define where data can live on your network, and the DLP solution will block ornotify you if that protected data is leaving the defined location. This may be useful, particularlywith a distributed source code system such as Git, to restrict where repository copies can becreated.HardeningHardening involves reducing the vulnerable surface of a server. Although these tasks aretypically OS changes, which fall under the domain of an OS or IT administrator, source codeadministrators need to understand the requirements of their tools and processes to guideeffective hardening. The hardening process looks at services running on the host and knownvulnerabilities. Remediation consists of techniques such as disabling and removing servicesnot needed to avoid unknown vulnerabilities, meeting minimum password requirements, and
6 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.installing the latest security patches. By reducing the vulnerable surface of a server, we makeit more difficult for an attacker to compromise it. For example, if you do not need Apache webservices on a server, disabling and removing the Apache services removes all attack methodsthat rely on Apache vulnerabilities. Work closely with the OS administrators because they maynot know which services the source code system requires.There are many hardening resources available to help secure your servers.Microsoft has had a variation of Windows Server Update Services since Windows 2000 SP3that enables automatic updates of the Microsoft OS to keep systems current with the latestpatches. Microsoft also provides several tools to understand and manage the security state ofyour servers:• Security Configuration & Analysis can compare your system against a security templatethat defines items such as Password Policy, Lockout Policy, User Rights, and RegistryPermissions (Microsoft, 2013). It comes with modifiable templates that were developedin conjunction with NIST, DISA, and NSA. You can learn more about NIST’s role on theNIST web site (NIST, 2008).• The Security Configuration Wizard will poll you for information about your environmentand how you will use the server to create a custom security template to manage yourserver in the context of the role it plays. The wizard can then apply the template(Microsoft, 2013).• The Microsoft Baseline Security Analyzer is an assessment tool that can analyze asystem for vulnerabilities such as missing security patches, common password issues,logging configuration, and some permission issues. It generates a report along withrecommendations to improve system security (Microsoft, 2013).The U.S. Defense Information Systems Agency, a part of the Department of Defense(DoD), also produces a set of Security Technical Implementation Guides (STIGs) that definethe DoD standards for hardening a wide variety of systems. You can analyze your systemsagainst STIGs manually or using a tool compliant with the security content automation protocolfor automated analysis. Combining the results from these scanning tools with an enterprisegovernance, risk, and compliance (EGRC) tool can help you track metrics to measurecompliance goals or furnish evidence for audits in the event of a security incident.Avoiding Malicious CodeThere are two situations to consider with malicious code:• Protecting your environment (servers, workstations, laptops) from malicious code, suchas malware that may be installed, and• Protecting against an attacker altering your source code in such a way that can harm acustomer environment through your product once it is installed.The first step for protection should be maintaining a strict chain of custody for everything youbring into your environment. Use only reputable sites to obtain components whether they areinternal development tools such as Eclipse or an open source component that you may use inyour product. Defining “reputable” can be challenging, but getting the latest JDK from Oracle orCNET is generally better than getting it from an unknown site.
7 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.With many open source components available on community sites such as SourceForge.net,you need to evaluate the reputation of the project as well. These sites often have informationabout the number of downloads for the component along with reviews and recommendations.This type of information must be evaluated with a critical eye, however; the numbers can bemanipulated and reviews can be forged, but it can be useful in narrowing the scope of projectsto review in depth. Detailed review should, at a minimum, analyze the development forum ofthe project, the number, quality, and frequency of patches, and the public bug repositorysystem as the tangibles that represent a “quality” of the project.Once you’ve settled on a reputable site, be sure to validate the integrity of your download byconfirming file size and checksums. Use cryptographic checksums (also known ascryptographic hash algorithms) such as SHA-2 or digital signatures in preference to otheroptions that might be available. Cryptographic checksums and digital signatures arespecifically designed to address challenges where common checksum algorithms, such asCRC-32, are easily reproduced with modified binaries. When possible, get the hashinformation from a different site or through a different medium; if the download site has beencompromised, the attacker can post checksum data that matches the corrupt package.After taking the effort to validate that you trust the component, keep a record of all of theinformation used to support your decision.Your next level of protection should come from anti-malware tools that scan your environmentand your product at various stages of the build process. While the merits of anti-malwaresoftware are being hotly debated, it does provide reassurance that known malware signaturescovered by the tool are being screened out.Finally, strong development processes that require peer code reviews, tracking of functionalrequirements for code branches, restricted commit access to the HEAD of the repository, anduse of code analysis tools reduce the likelihood of malicious code being inserted into yourcode base where it might ship to customers. Combined, these techniques help ensure thatonly intended changes are submitted to the mainline.LoggingLogs are an important forensic tool and must be carefully protected to preserve their integrity.Checking logs when there are no apparent problems with your environment can help youbetter recognize good and bad behaviors. By actively monitoring logs, you can watch forsuspicious behavior patterns such as log-in failures over a set threshold, code commits fromsomeone on vacation, or automation accounts being used from unusual locations.Start by confirming that your systems have logging enabled and make sure the logs areprotected. Restricting write access to the logs prevents alteration of the logs to hide attackactivity. Restricting read access can reduce leaking data, such as user IDs, that might beuseful in an attack scenario. You will also want to protect your logs from corruption due toinsufficient available storage through techniques such as log rolling and storage monitoring.Move (or at least copy) your logs to another system as frequently as possible. Local loggingusually can be trivially defeated by the same attacker with elevated privileges you are trying totrack. Moving the logs to another system with different privileges preserves the integrity ofthose logs.
8 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.Finally, consider a solution to analyze the content of your logs. The right tools can monitor logsfor predefined patterns, such as excessive log-in failures, and generate alerts to administratorsfor further investigation. More advanced searches can correlate information from multiple logsor other available security information. To correlate data across multiple sources, we needconsistent user IDs or a way to correlate the IDs.Next StepsOnce you’ve established a standard that reflects your expectations about protecting sourcecode, consider both a method to assess compliance and to track your progress. Whether youuse simple spreadsheets or an EGRC solution, you will want to measure, monitor, and reporton progress. The ability to measure compliance creates the feedback loop and visibility forcontinuous improvement.Understanding the EnvironmentWhile security initiatives are often supportive of typical SCM best practices, security and SCMdo not always speak the same language and need to find common ground. One challenge indeveloping a workable standard for your organization will be bridging this communication gap.Begin by building an understanding of your environment with both SCM and securitystakeholders.Both tools and processes contribute to the security of source code. Changing softwaredevelopment tools and processes is not something to undertake lightly. If a solution iscumbersome, developers will find ways to simplify their work, but that may unintentionallyundermine security controls. The goal is to understand how best to remediate securityvulnerabilities in your environment based on how your teams work and what risks you have.1. Consider the entire environment and its lifecycle in this analysis: your repositoryapplication, the host OS and hardware, storage, build environments, and backups. Athreat modeling exercise that covers the entire build environment will help youdetermine where weaknesses lie. A weakness anywhere in the environment canjeopardize your source code; consider these situations:• The Perforce application uses authentication tied to a centralized LDAP server, butthe underlying OS has local accounts with weak or default passwords; an attackerwho gains access to the host can modify your Perforce configuration.• Your source code repository is well secured, but your build hosts have a vulnerabilitythat allows an attacker to modify the build process to inject malicious code or changecompiler flags.• An attacker gains access to your unencrypted backup or a disk from adecommissioned source code environment. What could the attacker learn aboutyour products in the field or your next features?2. Compare the tools currently in use with your security requirements to determine if thereare significant issues with the tools themselves (e.g., they may not be capable ofencrypted sessions or using Active Directory). Often, the existing implementation maynot meet your security objectives, but you can change the configuration to support the
9 SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.requirements such as when an integration to your corporate Active Directory databaseis available but you’re not currently using it.3. Next, capture software development practices. Understanding how developers work willhelp you define the most transparent security solutions for the environment or allow youto defend practices that disrupt developers. A team that is used to having copies ofsource code on their laptops to work with at the airport is not going to like a solution thatrequires them to be connected to the company network in order to work; but if they’reworking on the crown jewels of the company, proper protection may requireinconveniencing developers.4. Finally, understand how security best practices are implemented in your tools.Generally, the security solutions natively implemented by the tool are less expensive toimplement and maintain than solutions that require additional plug-ins, integrations, orcustom-written code. One example would be integration with Active Directory forAuthentication and Access Controls using the vendor’s supplied integration. With add-ons such as plug-ins and scripts, almost any tool can be brought into compliance withmost security policies, but the trade-offs of initial cost, maintenance costs, availablesupport, and ease of use should be compared with migration costs to a solution thataddresses more of your primary concerns out of the box.Managing ChangeBased on the analysis of your environment, we now look to manage the implementation ofsecurity improvements.• Begin by minimizing new options: don’t chase a moving target. You’ve done theanalysis of what is and isn’t working in your environment, so narrow the scope ofacceptable solutions by recommending preferred solutions—at least until you canmanage what you have.• Provide strong guidance to encourage adoption of improved security techniques.Guidance should specify how to securely configure each solution to meet securityrequirements. Include security training focused on source code in addition to anyother organizational security training. The more developers buy in to the culture ofsecurity, the easier it will be to implement changes.• Develop an assessment framework to measure security compliance ofenvironments. If possible, use a compliance management tool to support theassessment and provide reporting. Use this framework to tie security compliance toprerelease metrics and create an environment of continuous improvement.• Create a framework to evaluate new solutions. When there is a new tool thatdevelopers want to use, this framework should help you quickly understand thesecurity considerations of the tool and provide guidance for compliance. Eventually,you should be able to use this framework to proactively supply security “reviews” ofnew tools.• Continue to learn about security and evolving threats.
10SOURCE CODE PROTECTION This is text for annotations in footer. Similar to footnotes treatment.ConclusionThe evolving threat landscape changes how we need to think about source code security.Evolving and advancing threats require more defense in depth to protect ourselves and ourcustomers. Expanding security awareness to SCM administrators and development teamsencourages better conversations about how to best protect source code.We need stakeholders to define policies and standards based on common security strategiesto set security goals. These documents provide reference points for establishing security andcommunicating improvements. Defense-in-depth techniques such as data classification,identity and authentication management, access controls, hardening, avoiding malicious code,and logging furnish a solid base from which to build environments with the right level ofsecurity for your organization.Through continuous analysis of our source code environments, we can assess risk and buildframeworks to support continuous improvements. These assessments and frameworkssupport prioritization of changes in a secure manner as well as provide assurance of the lastknown good state and strong forensic information when we face a potential security breach.Works ReferencedChickowski, E. (2013, January 24). Avoiding IAMs Biggest Blunder. Dark Reading.Department of Defense. (1995, January). DoD 5220.22-M National Industrial Security ProgramOperating Manual (NISPOM) January 1995. Retrieved February 2013, from USAID: DoD5220.22-M National Industrial Security Program Operating Manual (NISPOM) January 1995.FIPS PUB 199. (2004, February). Standards for Security Categorization of Federal Informationand Information Systems. Retrieved February 6, 2013, from NIST:http://csrc.nist.gov/publications/fips/fips199/FIPS-PUB-199-final.pdfKissel, R., Scholl, M., Skolochenko, S., & Li, X. (2006, September). NIST Special Publication800-88: Guidelines for Media Sanitation. National Institute of Standards and Technology.McGuiness, T. (2001). Defense In Depth. SANS Institute.Microsoft. (2013). Microsoft Baseline Security Analyzer. Retrieved 2013, from Microsoft:http://www.microsoft.com/en-us/download/details.aspx?id=7558#overviewMicrosoft. (2013). Security Configuration Wizard. Retrieved 2013, from Microsoft Technet:http://technet.microsoft.com/en-us/library/cc771492(v=WS.10).aspxMicrosoft. (2013). Step-by-Step Guide to Using the Security Configuration Tool Set. Retrieved2013 from Microsoft TechNet: http://technet.microsoft.com/en-us/library/bb742512.aspxNIST. (2008, October 10). CSRC - System Administration. Retrieved 2013, from NIST:http://csrc.nist.gov/itsec/Security for Business Innovation Council Initiative. (2011). When Advanced Persistent ThreatsGo Mainstream.The Apache Software Foundation. (2012). How the ASF Works. Retrieved February 2013,from The Apache Software Foundation: http://apache.org/foundation/how-it-works.html