CAMP: Content-Agnostic Malware Protection                     Moheeb Abu Rajab, Lucas Ballard, No´ Lutz, Panayiotis Mavrom...
Google’s Safe Browsing API shows for dangerous web sites.            Oberheide et al. [24] proposed CloudAV as a new model...
include content from malicious sites that automatically exploit
!                                                                         a browser with no interaction from the user. Thi...
In this paper, we make use of a binary analysis system provide as many features as are available to it. The followingthat ...
Fig. 2. The browser warns the user that the download is malicious. The   Fig. 3. The browser warns the user that the downl...
of the timestamp of the request to the URL. This causes the            a malicious reputation decision as well as the numb...
0.9                                                                                                                       ...
Request        Aggregate Index                         1 day                  7 day               28 day                  ...
analysis||urls                                       	          	                                           ...
measures how many different URLs under the domain
were analyzed by the binary analysis system and how often                    	                                            ...
they were labeled malicious, i.e. for the 7-day time period,the analyzer checked 20 URLs from and found one       ...
if CAMP has never seen it before and the binary is not signed    (Section III-C4). The despammer runs continuously, requir...
1                                                                                               1                         ...
Upcoming SlideShare
Loading in …5

Aburajab ndss-13


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Aburajab ndss-13

  1. 1. CAMP: Content-Agnostic Malware Protection Moheeb Abu Rajab, Lucas Ballard, No´ Lutz, Panayiotis Mavrommatis, Niels Provos e Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043 Abstract---In spite of recent advances, the world wide web re-is now in control of the computer and no vulnerabilities ormains an important vector for malware installation. Approaches exploits were required for the installation [28], [12].to evaluating potentially malicious code before execution in a Traditional defense mechanisms such as Anti-Virus (AV)browser, such as blacklisting or content-based detection are products are often ineffective against current malware down-hindered by an attacker’s ability to easily change hosting domainsor mutate malware binaries. On the other hand, whitelist- loads as AV engines tend to be content based and providebased approaches are challenged by the large, dynamic, and an oracle to malware authors allowing them to repackageheterogeneous space of benign binaries that they must track. In their software until it is no longer detected [24]. Similarly,practice these approaches continue to provide value for popular URL blacklists such as Google’s Safe Browsing API [16]binaries at either extreme of maliciousness (e.g., the current largeoutbreak of malware, the benign binaries shipped with an OS), work well for identifying compromised web pages which tendbut bridging the gap between whitelist and blacklist detection to be relatively static, but cannot be completely up to datefor web malware remains a significant challenge. in identifying all the domains from which malware is being This paper presents CAMP, a content-agnostic malware pro- distributed. The abundance of dynamic DNS providers allowtection system based on binary reputation that is designed to for frequent rotation of domains from which malware is beingaddress these shortcomings. CAMP is built into the browser anddetermines the reputation of most downloads locally, relying on distributed. If the domains rotate faster than the update intervalserver-side reputation data only when a local decision cannot be of the malware lists, domains distributing malware are moremade. This paper gives a detailed overview of CAMP and its likely to remain undetected [3], [27].architecture and provides an evaluation of the system through The malware arms race has shown that neither signaturea six-month deployment in which 200 million users of Google based AV engines nor malware lists are sufficient to protectChrome requested between eight to ten million reputation re-quests a day. Our evaluation shows that CAMP exhibits accuracy users. A whitelist based approach in which only trustedclose to 99% relative to proprietary VM-based dynamic analysis, software can be installed is more promising in that it canis able to process requests in less than 130 ms on average, and completely block the installation of socially engineered mal-was able to detect approximately five million intentional malwareware [6], [11]. Since such malware is not on the whitelist,downloads per month that were not detected by existing solutions.installation of it will not be allowed. However, whitelists suffer from drawbacks, too. The sheer variety of available software and the perpetual creation of new software makes the task I. I NTRODUCTION of creating a whitelist that covers all benign software nearly Malware is a popular venue for monetization in the un- impossible. As a result, neither whitelist nor blacklist basedderground economy [7]. Adversaries frequently compromise approaches provide sufficient coverage to protect users fromvulnerabilities in users’ machines to automatically install mal- malicious software in practice.ware via so-called drive-by downloads. The action of visiting This paper offers a different approach in which the gap be-a malicious web page in a vulnerable browser is sufficient tween known benign and known malicious software is bridgedfor an adversary to gain complete control over the vulnerable by a content-agnostic reputation-based detection approach.machine allowing her to install arbitrary software. Frequently, CAMP, our detection system, consists of a client componentinstalled malware can cause the computer to become part of built into Google Chrome and a server component responsiblea botnet or can exfiltrate sensitive data such as credit card for maintaining a reputation system that predicts the likelihoodnumbers to be sold in the underground economy [20]. that a downloaded binary is malicious. CAMP makes use As browsers and plugins are becoming more secure, the of Google’s Safe Browsing API to detect downloads knownvulnerable user base is shrinking and drive-by downloads to be malicious. For all binaries not detected, we extractare becoming less effective as a malware distribution vector. additional features from information available to the webWhile drive-by downloads are still a popular distribution vec- browser about the download and use these features to build ator, adversaries are increasingly turning to social engineering server-based reputation system. The reputation system predictsfor malware distribution. Fake Anti-Virus products are one if an unknown binary is malicious without prior knowledge ofsuch example in which the user is led to believe that their its content. We built CAMP to protect users of Google Chromecomputer has been infected and a free product to remedy the from downloading malware on Windows. Our system protectssituation is offered. Consequently, the user voluntarily down- over 200 million users, makes millions of reputation-basedloads and installs malware under the guise of a free security decisions every day, and identifies about 5 million malwaresolution. The benefit to the adversary is that the malware downloads every month beyond the malware warnings that
  2. 2. Google’s Safe Browsing API shows for dangerous web sites. Oberheide et al. [24] proposed CloudAV as a new modelCAMP achieves an accuracy close to 99% relative to propri- to increase the effectiveness of Anti-Virus products. Underetary, VM-based dynamic analysis. this model, Anti-Virus protection is moved into the cloud and In this paper, we provide a detailed overview of the design multiple AV engines, working in parallel, are employed toand architecture that comprises our reputation-based detection improve detection rates. While combining the decision of mul-system and evaluate its effectiveness based on data collected tiple AV engines leads to better detection, this approach is stillfrom February to July 2012. We discuss how the system subject to the limitations mentioned above as CloudAV alsominimizes the impact on the privacy of browser users and relies on timely signature updates. Furthermore, Oberheide etevaluate its detection performance focusing especially on false al. show that major Anti-Virus engines detect only 35% to 70%positives. of recent malware which means that many malware binaries This paper makes the following contributions: remain undetected. CloudAV also requires that all binaries • We present CAMP, a content-agnostic malware protec- are uploaded to the cloud which exposes these downloads tion system, which utilizes reputation-based detection to to a third-party and may constitute an unacceptable loss in protect users from web-based malware. privacy for some users. While CAMP also moves detection • We perform an extensive, six month evaluation of CAMP of malware into the cloud, it reduces the privacy impact by consisting of over 200 million unique users and millions employing whitelists so that most download URLs stay within of daily reputation evaluations. We show that our content- the browser and do not need to be sent to a third party. Binary agnostic detection approach is both accurate, with an payloads never leave the browser. Additionally, its content- accuracy close to 99%, and well performing, processing agnostic approach does not suffer from the same limitations requests in less than 130 ms on average. as AV engines, e.g. delays in signature updates. • We compare CAMP with the current state of practice b) Blacklist-based protection: Blacklist-based and demonstrate that CAMP outperforms Anti-Virus, as approaches provide protection from malware by identifying well as various web services, e.g. McAfee’s Site Advi- the sites from which it is being served. Services such sor, Symantec’s SafeWeb, etc. Further, CAMP identifies as Google’s Safe Browsing API [16], McAfee’s Site large numbers of malware undetected by these services, Advisor [22] or Symantec’s Safe Web [23] detect malicious including 5 million malware downloads per month not or compromised web sites that may infect users with malware. identified by Google’s Safe Browsing API. Browsers integrate with these services directly or via tool bars and prevent users from visiting known malicious sites. The remainder of the paper is organized as follows. We While effective in identifying compromised web-sites whichdiscuss related work in Section II. In Section III, we present tend to be long lived, blacklists face challenges when tryinga detailed description of the system architecture and all to protect against adversaries that employ highly agilecomponents responsible for creating the reputation system. malware distribution servers [27]. By frequently changing theSection IV discusses how to leverage the reputation system domains that serve malware, blacklists become less effectivefor malware detection. We evaluate the performance of the as domains rotate faster than the time it takes to detect them.system in Section V, and present a small case study of a highly CAMP offers a different approach. Instead of exportingdynamic malware campaign. We discuss CAMP’s unique a blacklist to clients, CAMP protects users by augmentingproperties in Section VI. Finally, we conclude in Section VII. blacklists and whitelists with a content-agnostic reputation system that protects users from malicious binaries without II. R ELATED W ORK requiring a-priori knowledge of the binary or its serving Protecting users from malware continues to be a challenging domains. Specifically, the browser performs the following toproblem. In the following, we provide an overview of differ- check the safety of a downloaded file: (1) the browser triesent approaches to prevent malware from infecting users and to locally determine whether the download is malicious byexplain how CAMP differs from prior work. checking the download URL against a list of URLs known a) Content-based detection: Anti-Virus software is one to serve malware exported to the browser using Google’sof the most common content-based malware detection tech- SafeBrowsing API, (2) the browser checks locally againstniques. Usually, Anti-Virus products run on end-user systems a dynamically updated list of trusted domains and trustedand employ signature-based detection to identify variants of binary signers to determine whether the downloaded content isknown malware. likely benign and, (3) for downloads that do not match any of While Anti-Virus engines do help in protecting users from the local lists, the browser extracts content-agnostic featuresmalware infections, several challenges limit their effectiveness. from the download and sends a request to CAMP’s reputationMalware authors have developed increasingly sophisticated service. As a result, CAMP protects users against maliciousevasion techniques, such as packing and polymorphism, aimed downloads that would likely have been missed by a blacklist-at circumventing detection by AV engines [4], [30]. Addi- based approach. We compare CAMP’s performance to populartionally, the signature generation and update cycle causes an blacklist services in Section V.inherent delay in protecting users against new variants of CAMP and Google’s SafeBrowsing API complement onemalware. another. The API exports blacklists of compromised sites that
  3. 3. include content from malicious sites that automatically exploit
  4. 4. ! a browser with no interaction from the user. This works wellfor protecting browsers from infected landing pages as theyare relatively static, and compromise can be observed by an automated system to build the blacklist [25]. CAMP, on the other hand, targets binaries that were downloaded by misledusers from highly dynamic infrastructures. c) Whitelist-based schemes: In contrast to blacklists, whitelists ensure that only known benign software can beinstalled and installation of everything else is disallowed. Forexample, Bit9 [6] and CoreTrace [11] maintain a list of bina- ries verified to be benign. While whitelisting can be effectivein enterprise settings, it remains challenging to maintain anup-to-date whitelist that covers the plethora of applications developed globally. Therefore, protecting downloads in thebrowser through whitelists alone is not currently practical. Fig. 1. The diagram presents a high-level overview of the detection systemNonetheless, to protect user privacy, CAMP derives a whitelist showing the communication between the client and server as well as thefrom its reputation data that is used by the web browser server-side processing locally decide if a binary download should be trusted.Downloads not on the whitelist require a reputation-baseddecision. malware is installed in the background without knowledge d) Reputation-based detection: A lot of research has by the user. We believe that focusing on user downloadsbeen conducted on reputation systems [19] and how to use is justified for several reasons: the security of browsers hasthem for detecting malicious activities. Hao et al. [18] pro- increased significantly over the last few years [17], [29], [32]posed SNARE, a reputation system for detecting spam email. and proposed exploit detection systems such as Blade [21]Qian et al. [26] proposed using network-based clustering to or Zozzle [13] offer efficient mechanisms to prevent currentincrease the accuracy of spam-oriented blacklists. exploits from succeeding. As a result, automated exploitation Notos [2] and EXPOSURE [5] offer a dynamic reputation of browsers has become more difficult and adversaries are alsoengine for DNS. Both systems use features based on passive incorporating social engineering techniques in which usersDNS resolution to predict the likelihood that a domain name is download malware themselves.malicious. CAMP is complimentary to Notos and EXPOSURE To efficiently determine if an unknown binary downloadbut is specifically targeted to predict whether an unknown is malicious, CAMP is split into a client component and abinary download is malicious. Furthermore, the use of DNS server component. The client component is integrated into alimits a reputation system to making decisions on the granu- web browser. As a first step, the client checks the downloadlarity of domain names whereas CAMP operates on all of the against both a local blacklist and whitelist. If no match isfeatures that are associated with a downloaded binary. found, the client extracts features from the binary download, Finally, closely related to our work, is Microsoft’s sends these features to the server, and displays a warning toSmartScreen described briefly in a blog post [10]. SmartScreen users if the server response instructs it to. The server processesutilizes a reputation-based approach to protect users from the features sent from the browser and computes a reputationsocially-engineered malware. It does so by computing reputa- decision informed by the client request and a reputation metriction for the download URL, the file identifier and the publisher constructed from previous client requests. Figure 1 shows anif available as a digital signature. Unlike CAMP, it requires overview of the complete detection system. We will discussthat all download URLs are sent to a remote server. In contrast it in detail in the following SmartScreen, CAMP computes its reputation based on manyother features available to the browser, such as referrer URLs, A. Binary AnalysisIP addresses, etc. In addition, this paper provides in-depthtechnical details of CAMP’s reputation system and evaluates Many machine learning algorithms require labeled groundits performance in an operational setting. truth for training purposes. Our situation is similar in that we need to label our reputation data according to the nature of III. S YSTEM A RCHITECTURE the binary, e.g. whether it is malicious or benign. We support In the following, we provide an overview of CAMP’s design multiple dimensions for the labels so other classifications suchand architecture. Our goal is to create a system that scales as spyware are possible, too. Similarly, the type of binary is itsto hundreds of millions of users and protects them from own dimension, e.g. Windows PEbins are treated differentlydownloading malware while at the same time limiting the from Mac OS X DMG files. However, for any binary type,impact on their privacy. Our focus is specifically on binaries a corresponding classification system needs to be available todownloaded by users rather than drive-by downloads in which provide accurate labels.
  5. 5. In this paper, we make use of a binary analysis system provide as many features as are available to it. The followingthat we developed independently to classify binaries based on is a list of features usually available to web browsers:static features as well as dynamic execution traces. The main • The final download URL and IP address correspondinggoal of this system is to limit the number of false positives to the server hosting the download.and we consciously sacrifice detection in favor of fewer false • Any referrer URL and corresponding IP address encoun-positives. tered when initiating the download, e.g. the results of The labels produced by the binary analysis system form the multiple HTTP redirects.ground truth that governs the training of the reputation system. • The size of the download as well as content hashes, e.g.The labels also allow us to compute detection performance of SHA-256.the reputation system post facto. We measure this as part of • The signature attached to the download which includesour evaluation in Section V. the signer as well any certificate chain leading to it. The It is important to note that our reputation system does not client also informs the server if it could successfullyrequire a particular binary classification solution and other verify the signature and if it trusted the signature, e.g.detection approaches [9], [33] or labeling solely based on the it was rooted in a trusted certificate authority.output of AV engines should also be possible. Of course, the The client then sends these features to the server and awaitsoverall accuracy of the system would be dependent on the its response. The server may reply with several differentaccuracy of the underlying labeling mechanism. verdicts based on the reputation data available to it. It can Binary analysis systems are not perfect and are susceptible inform the web browser that the download is predicted to beto evasion [15], [31]. Since CAMP is a reputation system benign in which case the web browser completes the downloadthat requires a reasonable estimate of ground truth, any errors without displaying a warning, or it can tell the web browserin labeling might propagate to CAMP’s decisions. However, that the download is either deemed malicious (Figure 2) orsince CAMP is independent of the classification mechanism, unknown (Figure 3). In both of the latter cases, the webany advances to dynamic analysis or Anti-Virus products will browser warns the user about the download and offers to deleteseamlessly improve CAMP’s performance. Nonetheless, we the downloaded file. The unknown verdict indicates that theshow in Section V-B that our binary analysis system exhibits server did not have sufficient reputation information to labelreasonable accuracy. the download as either benign or malicious. Our evaluationB. Client shows that in the majority of all cases unknown by itself When a user initiates a binary download in her web browser, is good a predictor for malicious downloads. The differentthe web browser applies several checks before asking the reputation verdicts are explained in more detail in Section IV.server for a reputation decision. If any of these checks fail, User privacy is an important goal for CAMP. Verifying thethe client makes a local decision on whether the download is content type of the file and that it neither matches blacklistsmalicious or not. nor whitelists drastically limits the number of downloads for 1) The web browser determines if the binary is already which a remote server is contacted. As shown in Section III-D, known to be malicious based on its URL, for example, the web browser contacts the CAMP service for only about via a list of known malware sites. In our case, we use 30% of binary downloads. Furthermore, the web browser sends Google’s Safe Browsing API to make that determination. only features computed from the binary, not the binary itself, If the binary is known to be malicious, the browser can to the server. display a warning directly without requiring a reputation decision. C. Server 2) The browser determines if the binary download could The server pipeline has two different roles when processing potentially be harmful to the computer, e.g. it might requests. First, the server receives the client request and correspond to an executable which may carry malicious renders a reputation verdict based on its reputation system code, or a DMG which is how Mac OS X software is which encompasses aggregate information from all downloads installed. observed by the server during its measurement intervals, 3) The binary download is checked against a dynamically including e.g. 1 day, 7 day and 90 day intervals. Second, the updated whitelist of trusted domains and trusted binary server uses the information provided by the client to update signers. The list of trusted domains or trusted signing its reputation data. certificates consists of reputable software publishers The reputation verdict is computed by a reputation metric known for not distributing malware. If the binary down- calculated by a binary circuit that has access to all features load matches the whitelist, the browser assumes that from the client request and any reputation data that is ref- the download is benign and no server-side reputation erenced by those features, e.g. how many known benign or decision is required. malicious binaries are hosted on a given IP address, etc. If all the above checks do not result in a local decision, the To incorporate information provided by the clients intobrowser extracts features from the downloaded binary. Since the reputation system, client requests are first despammed tothe reputation decision is made on the server, the client can prevent misbehaving clients from unduly influencing the data.
  6. 6. Fig. 2. The browser warns the user that the download is malicious. The Fig. 3. The browser warns the user that the download is not commonly intentionally discrete arrow presents an option to keep the file. downloaded.Despammed requests are then processed within a few minutes diminish its generate up-to-date features. As CAMP is deployed to a large number of users, many To create a classification system that has both high per- of the design decisions in building the system are in favorformance and high reliability, we employ BigTable [8] and of reducing false positives. However, the performance of theMapReduce [14] for distributed data storage and parallel reputation system can be adjusted gradually to favor recallprocessing. In the following, we provide a detailed overview over precision.of each component involved in making reputation decisions. We provide a detailed discussion of CAMP’s reputation- 1) Reputation System: The heart of the decision process is based detection in the next section but give an overviewthe reputation system. To better understand its properties, we here of the reputation system itself. The reputation systemplace it within the analysis framework proposed by Hoffman is responsible for receiving a browser reputation request andet al. [19]. According to Hoffman a reputation system can be replying to it with a verdict.characterized across the following three dimensions: For each client request, the reputation system can make a decision based on a-priori information if either the URL or the • Formulation, which represents the mathematical underpin- content hash is known to be malicious. Similarly, to respond to nings of the reputation metric as well as its information major false positives, the reputation system consults a server- sources. side whitelist to override any reputation decision. • Calculation, which is the concrete implementation of the The reputation metric is calculated by a binary circuit that formulation. references reputation data in form of previously computed • Dissemination, which characterizes how the results from aggregates. The features from the client request and the the reputation metric are propagated to participants. reputation formulation determine which aggregates are looked In our case, both the calculation and dissemination are up from the data store. The reputation system then computescentralized and deterministic. The storage of reputation data a verdict which can be either: benign, malicious or unknown;is transient as each item expires after 90 of days. see Section IV for a discussion of the different meanings. The In the following, we explain the formulation of the reputa- data store lookups happen in parallel and are non-blocking totion system in more details. reduce overall latency. The time spent computing the decision Our reputation data is based solely on direct, automatic from the aggregates is insignificant compared to the time itsources. The output of the binary analysis pipeline is a trusted takes to look up data from the data store.automatic source. Data collected and sent by web browsers 2) Frontend and Data Storage: The frontend is responsibleis also a direct, automatic source but may be untrusted. for receiving requests from web browsers and answering themThe reputation data consists of features across a variety of without incurring any significant latency. To achieve lowdimensions that each describe some aspect of the binary or latency, we split the reputation metric computation and theits hosting infrastructure. As mentioned in Section III-B, for integration of new data into the reputation system into separateeach binary download, we receive not only a content-hash but components. Upon receiving a request, the frontend issues aalso related information such as corresponding URLs and IP Remote Procedure Call (RPC) to the reputation system, whichaddresses of the servers hosting them, etc. The server may determines whether the binary download is malicious. Afterderive further features from the client request. receiving an answer, the frontend writes the request and the The reputation data maps each feature to an aggregate that verdict to a data store that other components of the pipelinecontains measurements over data observed during a given can process, and then returns the verdict to the client.time frame. Aggregates are continuous features and consist As CAMP needs to handle a large number of web browserof two counts: the number of interesting observations and the requests, the temporary storage of request data requires atotal number of observations. For example, assume CAMP carefully chosen storage layout. We use BigTable [8], a non-observed 10 downloads, 6 of which were malicious, on IP relational distributed database that provides key-value storesaddress IPA . The aggregate corresponding to the feature and allows the association of a timestamp with a given key.IP:IPA would then be {6, 10}. Each aggregate also contains While Bigtable scales well, it is limited to approximately 2GBthe first and last time the particular feature was seen. of data per key. For subsequent data processing it is helpful The aggregates include both positive as well as negative to index requests by the URL of the binary. However, asevents, i.e. they can be both trust building and trust diminish- we store each request for two weeks and some URLs areing. For example, the number of users downloading a binary requested frequently, on the order of hundreds of thousandsfrom a site may represent an increase in trust. On the other times a day, we chose not to index solely by URL. Instead, wehand, the number of malicious binaries hosted on a site may append the Reverse-Ordered hexadecimal string representation
  7. 7. of the timestamp of the request to the URL. This causes the a malicious reputation decision as well as the numberdata to be placed in different rows while maintaining identical of requests for binaries known a priori to be malicious,ordering compared to indexing by URL. This design decision either based on their URL or corresponding contentwas crucial in scaling CAMP to handle popular URLs. hash. For the aggregates from the binary analysis system, 3) Spam Filtering: The spam filter processes the data writ- this contains the number of URLs hosting maliciousten by the frontends in real time and discards requests that do downloads as well as the number of malicious contentnot originate from regular users. We do not require the spam hashes.filter to be highly accurate and false positives are acceptable. For example, the aggregate forThe spam filter may make use of any information provided client||reputationin the client request and has visibility into all client requests represents the total number of client requests for the sitemade within the last 24 hours. As a result, the spam filter can as well as the number of client requests for the sameapply velocity controls on the user IP address of the request, site that received a malicious reputation decision. Anotherthe Autonomous System Number (ASN) corresponding to the example isIP address, etc. The spam filter also ensures that requests are analysis||urlsproperly formed and contain all required features, e.g. properly which contains the total number of URLs found under foo.comformatted URLs, etc. as well as the number of such URLs that were labeled Requests not discarded by the spam filter are forwarded malicious by the binary analysis an aggregator that computes aggregate features used by To construct these aggregates, the despamming componentthe reputation system. The output of the spam filter is also writes client requests to a temporary data store which isemployed to fetch binaries from the web that have not been indexed by the second aggregator index dimension. Then aanalyzed by the binary classifier. Since binary downloads may series of MapReduces [14] periodically processes all entriescarry sensitive information, we apply further filters so that in the data store. This process merges new data with olderonly binaries that exhibit sufficient diversity of context are aggregates to generate aggregates over different time intervals,considered for analysis. The binary analysis does not gate currently 1 day, 7 days, 14 days, 28 days and 98 days. Allany reputation decision and may complete a long time after a aggregates computed this way are available to the reputationreputation decision was sent back to the web browser. system. As stated previously, our goal is not only to make highlyaccurate decisions but also to reduce the impact on the privacy D. Client-side Whitelistof web users as much as possible. The requests received from CAMP reduces the privacy impact on its users in severalweb browsers reveal not only the binary URL visited by a user different ways. Since the detection is content-agnostic, webbut by their very nature also a corresponding source IP address. browsers do not need to send binary payloads to CAMP’sThe data processing architecture of the reputation system is servers. As users download binaries that potentially containdesigned so that the source IP address is visible only to the sensitive information, any solution that requires transmissionspam filter and completely deleted after two weeks. The binary of payload data is likely to face privacy challenges.URL is visible to rest of the pipeline, but is stored in such Another way in which CAMP reduces the privacy impacta way that it also is automatically deleted after two weeks. is by client-side whitelists that determine which binaries areThe only information that is stored for up to 90 days are the trusted in advance. If a binary is trusted, CAMP does not needaggregates that make up the reputation data. to send the binary URL to a remote server. The whitelists 4) Aggregator: The basis of the reputation system is contain trusted domains such as as well asformed by the reputation data which consists of statistical trusted software publishers. A software publisher is identifiedaggregates indexed by keys derived from request features. by the public key of the signing certificate and the CA certi-Aggregates are computed and written by the aggregator, which fying it. The goal of the whitelists is to resolve the majorityprocesses the output of the despammer and organizes aggre- of all downloads locally without requiring a reputation-basedgates according to a three-dimensional index. decision. At the time of this writing, approximately 70% of all • The first index dimension is defined by whether the downloads are considered benign due to policy or matching aggregate is computed from client requests or based on client-side whitelists. aggregates from the binary analysis system. Aggregates For a domain or signer to be added to the whitelist, it needs computed from client requests are considered untrusted to fulfill the following criterion. CAMP needs to have seen the whereas aggregates from the binary analysis system are binaries from the domain or signer for at least 90 days without inherently trusted. encountering any signs of maliciousness. We add new domains • The second index dimension consists of features from or signers to the whitelist in the order in which they contribute client requests and additional features derived from the to the number of requests received by CAMP over that time request on the server side. period. • The third index dimension contains broad categories over Since adding a domain or signer to the whitelist implies which aggregates are computed. For client side aggre- that the CAMP server no longer receives any downloads for it, gates, this contains the number of requests that received we employ a web crawl to regularly analyze binaries hosted
  8. 8. 0.9 determine with a high degree of confidence whether a binary 0.85 is benign or not. Binaries hosted on the web fall within a wide spectrum rang- 0.8 ing from known-benign to known-malicious. Known-benign % of whitelisted download requests. 0.75 binaries can be added to a whitelist and known-malicious 0.7 binaries to a blacklist. In an ideal situation, any binary is either known to be good or known to be bad. The reputation 0.65 system described here is concerned with the gray area between 0.6 whitelists and blacklists. For CAMP, we attempt to create a whitelist that covers the 0.55 majority of benign binaries and subject anything not covered 0.5 by it to a reputation decision. In the following, we explain Signers Domains how to classify a binary using reputation data. 0.45 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Whitelist size. A. Feature ExtractionFig. 4. The graph shows the percentage of downloads for which a web For each feature in the client request, the reputation systembrowser would make a local reputation decision depending on the size of thedomain and signer whitelist. derives features that form the indices for aggregate lookups. The aggregates contain statistics about the total number of times a feature occurred and how many times it occurred in a malicious context. Each feature type has a different set ofon the trusted domains or signed by trusted signers. If the derived features.binary analysis reveals evidence of malicious behavior, the For IP addresses, the features are the IP address itself, thecorresponding entries are removed from the whitelist and are corresponding /24 netblock, as well as the corresponding /16again subject to reputation-based detection. netblock. The rationale for using netblocks is that serving IP Figure 4 shows the number of downloads that a web address frequently change, e.g. due to load-balancing or due tobrowser would handle locally when adding more trusted sites adversaries switching IP addresses to avoid IP-based the domain or the signer whitelist. We estimated these For netblocks that are used predominantly for serving mali-values by considering all reputation requests that we received, cious content, the likelihood that a yet unknown IP address inthen iteratively adding the most popular trusted signers and that netblock is also serving malicious content is often to our whitelist and tracking which requests would CAMP also uses URL-based features. The reputation re-have been handled locally by the browser. The graph does not quest from the client contains the URL pointing directly tocontain measurements for whitelists smaller than our initial the binary download as well as any URL encountered by thewhitelists as CAMP does not have any knowledge about web browser when following redirects to the binary. For eachrequests corresponding to them. Our initial domain whitelist URL, the reputation system extracts the host, the domain, andcontained 243 entries and matched approximately 47% of all the site. The domain and site are usually identical, but differdownload requests remaining after policy checks. The graph for dynamic DNS providers where the site is the same as theshows that by increasing the domain whitelist to 1000 entries host name. The goal of using the site and domain as a featurethe local percentage of downloads not requiring a request to is to track potential ownership of resources hosted on them.CAMP would increase to 73%. The signer whitelist is more For the signature-based features, we extract the signereffective as increasing it to 1000 entries would cover about key and the corresponding CA certifying the key for each85% of downloads locally. certificate chain encountered in the binary signature. We also keep track if the signature was trusted on the client, e.g. IV. R EPUTATION S YSTEM the client had a trusted CA that was the root of one of the The goal of our reputation system is to determine a-priori if certificate chains.a binary is likely going to be malicious or not without having Some of the client features such as the content hash areaccess to the content of the binary itself. Instead, metadata used directly in subsequent aggregate lookups.about the binary such, as its hosting infrastructure and howthe user reached it, form the basis for detection. B. Aggregate Computation One of the challenges with detecting malware is the ease The aggregation step discussed in Section III-C4 computeswith which malware authors can change their binaries, e.g. aggregates for all possible dimensions based on client featuresby repacking, and how quickly they can change the locations and features derived on the server. However, not all of thesefrom which binaries are being served. On the other hand, aggregates are consulted when making a reputation decision.there is no need to frequently change the hosting location of Which aggregates are consulted for rendering a reputationbenign binaries or the binaries themselves. Differences such as verdict depends on the configuration of the reputation systemthese may be leveraged to create a reputation metric that can described below.
  9. 9. Request Aggregate Index 1 day 7 day 28 day URL analysis——urls 0/1 1/10 3/100 analysis——urls 0/2 1/20 7/200 analysis——digests 3/22 4/25 11/153 client——requests 10/20 20/50 123/2200 IP analysis—ip:—urls 0/1 0/1 0/1 analysis—ip:—digests 0/1 0/1 0/1 client—ip:—requests 1183/1208 9801/11327 33555/37455 analysis—ip24:—urls 2/112 5/237 6/238 analysis—ip16:—digests 0/3 0/5 0/5 client—ip24:—requests 1313/2571 9912/32227 34127/43119 TABLE IT HIS TABLE SHOWS HOW REQUEST FEATURES RELATE TO DIFFERENT INDICES AND CORRESPONDING REPUTATION AGGREGATES . F OR BREVITY , NOT ALL CLIENT FEATURES NOR ALL DERIVED FEATURES ARE SHOWN HERE . Table I provides an example of how client features such as an IP address or URL are turned into indices for lookingup aggregates. The counts associated with each index measurehow often the index was encountered either in client requestsor by the binary analysis pipeline. The first value counts the number of malicious verdicts and the second value counts thetotal number of occurrences. For example, the index
  10. 10. analysis||urls
  11. 11. measures how many different URLs under the domain
  12. 12. were analyzed by the binary analysis system and how often
  13. 13. they were labeled malicious, i.e. for the 7-day time period,the analyzer checked 20 URLs from and found one Fig. 5. This figure shows a depth-2 boolean circuit for classifying a binaryof them malicious. based on boolean inputs computed from the aggregates of a reputation request (‘‘Ping’’). An example for a client aggregate is the index client|ip24:|requestswhich measures how many client requests for the netblock10.0.0.1/24 were received by CAMP during the specified time boolean circuit and the thresholds for each aggregate. Theseperiod. For the last 24 hours, CAMP received 2571 requests thresholds are determined via a training phase discussed and used reputation to label 1313 of them as malicious. Logically, the boolean circuit can be separated into three As the binary analysis system provides us with ground truth different parts: the site aggregates, the IP aggregates and thefor client requests for which the payload is known to us, we unknown aggregates.make decisions based on the direct and derived features as For the IP and site aggregates, we use two different func-well as their corresponding aggregates. We explain how the tions to map an aggregate to a boolean value. The threshold pdecision is calculated in the next section. function is defined as f (p, n, t) := n ≥ t and the count function is defined as f (p, n, t) := n ≥ t where p is theC. Reputation Decision number of positive samples in the aggregate, n is the total number of samples and t is the threshold provided by the The aggregates computed for each reputation request pro- model.vide a large number of continuously valued features suitable The booleans derived from the unknown aggregates arefor many types of machine learning. One drawback of em- more complex. The goal of the unknown aggregates is to flagploying machine learning is that the trained models are often binaries for which we do not have sufficient reputation tolarge and difficult to understand by humans. As any detection label them benign. The input is formed by the following threemechanism is going to produce false positives, we favor a negated booleans:classifier that allows human inspection and reasoning aboutfalse positives. To achieve this, we employ a depth-2 boolean 1) The boolean is true if the binary analysis pipeline hascircuit with a small number of boolean inputs. Figure 5 shows already analyzed the binary based on the content hash.the boolean circuit and its inputs as used in CAMP. Its 2) The boolean is true if the binary was signed and thestructure is simple as it contains only AND gates at depth 1 and client could trace the certificate chain to a trusted CAone OR gate at depth 2. Each individual AND gate represents root.a single detection rule. In the following, we use AND gate and 3) The boolean is true if the binary is popular because arule synonymously. large number of users downloaded either the digest itself Aggregates are mapped to boolean inputs via simple thresh- or other binaries from the same hosting site.old functions. The reputation metric is calculated from a As a result, a binary is labeled unknown by the binary circuit
  14. 14. if CAMP has never seen it before and the binary is not signed (Section III-C4). The despammer runs continuously, requiringby a trusted signer and not from a popular download site. 1.25 cores and 25 GB RAM. The valid requests that theThe parameters are thresholds on the number of downloads despammer outputs are processed by an indexer in the aggre-required before the binary or the site is considered popular. gation pipeline, which writes data to a temporary BigTable [8] The reputation system is configured by choosing thresholds for aggregation by a series of MapReduces [14]. The indexeraccording to the precision and recall for each AND gate. uses 2 cores and 8 GB RAM. The aggregation MapReducesPrecision and recall are determined from a labeled training run periodically throughout the day. Each run finishes inset. The training set is created by matching the content hash approximately one hour and uses 90 cores and 120 GBincluded in reputation requests from web browsers with the RAM. The MapReduces produce a model that is comprised ofcontent hash from the binary analysis system. The binary approximately 4.75 billion aggregates requiring approximatelyanalysis system provides the label, e.g. benign or malicious, 1.4 TB of BigTable storage. This model is replicated toand the reputation request provides the features. The training locations near each of CAMP’s frontends.set consists of 4, 000 benign requests and 1, 000 malicious The above metrics do not include resources for running ourrequests. binary analysis pipeline, or the CPU and memory resources Figure 6 shows the precision and recall for site aggregates required by BigTable.that result from changing the thresholds for the aggregatemapping functions. We see that achieving a precision larger B. Accuracy of Binary Analysisthan 0.95 results in recall of less than 0.2. Before we evaluate CAMP’s accuracy relative to our base- Figure 7 shows the precision and recall for IP aggregates. line proprietary VM-based dynamic analysis framework, it isWe see that achieving a precision larger than 0.95 results in important to understand the accuracy of that framework. Torecall of less than 0.15. do so, we compared the framework against the AV scanners We omit the precision and recall for the unknown rule as provided by VirusTotal [1].they are very similar to the previous two graphs. As can be We selected a sample of 2200 binaries that our dynamicseen from these graphs, a high precision leads to low recall analysis framework processed on a single day, but were notwhen considering each AND gate individually. Our evaluation known to VirusTotal. Of these, 1100 were labeled clean by ourin Section V shows that the combination of all AND gates, that dynamic analysis framework, and 1100 were labeled the complete boolean circuit, leads to acceptable detection We submitted each of the binaries to VirusTotal, and waitedresults with very low false positive rates. 10 days, to allow AV engines to catch up. We then consulted V. E VALUATION VirusTotal for each of the 2200 binaries to see how many AV engines agreed with our initial analysis. For CAMP’s initial deployment, we used Google Chrome After 10 days, 99% of the binaries that were flagged asand targeted Windows executables, e.g. PEbins, MSI-files, etc.. malicious by our framework were flagged by 20% or moreThe following evaluation measures the performance of CAMP of AV engines on VirusTotal. Only 12% of the binaries thatin that context. We evaluate resource requirements, accuracy we flagged as clean were also flagged by 20% or more of theof the baseline dynamic analysis framework, and the precision AV engines. Of these false negatives, a large percentage areof the reputation system. We also provide a comparison to classified as AdWare, which our system intentionally does notother common malware prevention solutions and a case study detect.of a campaign that CAMP identified. These results indicate that, by design, our dynamic analysisA. Server-Side Resource Requirements framework has a very high true positive rate, but does suffer from some false negatives. As we improve this framework, Here we briefly overview CAMP’s operational performance CAMP’s detection rates should improve as well.and resource requirements. To service 200 million GoogleChrome users, CAMP frontends and reputation RPC servers C. Accuracy of CAMP(Section III-C2) use approximately 4 cores and 9 GB RAMglobally to serve on average 190 QPS. Most of these resources The evaluation is based on data collected between Februaryare reserved for geographic distribution and load balancing, 2012 to July 2012 from our production deployment of CAMP.and we have load-tested CAMP frontends at 400 QPS using During that time frame, CAMP was used by approximately1 core and 3 GB RAM. The median latency is 127 ms and 200 million users. Each day, it received between 8 million tothe 90th percentile is 313 ms. As many binary downloads 10 million reputation requests and labeled approximately 200take much longer time, on the order of seconds, requesting to 300 thousand download requests as malicious; During oura CAMP verdict does not significantly add to the download evaluation, the reputation database kept track of approximatelylatency observed by users. 3.2 billion aggregates. Download requests can be processed with minimal overhead To get a better understanding of how well the reputationsince CAMP leverages preprocessing to build models that metric works in practice, we measured the true positive (tpr),are amenable to fast lookup. This is a multi-stage process, false positive (fpr), true negative (tnr) and false negative ratesencompassing despamming (Section III-C3) and aggregation (fnr) of the reputation requests received by CAMP. The rates
  15. 15. 1 1 bad binaries thr. bad binaries thr. bad requests thr. bad requests thr. bad URLs thr. bad URLs thr. 0.8 0.8 0.6 0.6 Recall Recall 0.4 0.4 0.2 0.2 0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Precision Precision Fig. 6. The figure shows precision and recall for site aggregates for Fig. 7. The figure shows precision and recall for IP aggregates for different different thresholds of the aggregate mapping functions. thresholds of the aggregate mapping functions. 1 0.05 True Positive True Negative False Positive False Negative 0.8 0.04 0.6 0.03 Rate Rate 0.4 0.02 0.2 0.01 0 0 03 04 05 06 07 03 04 05 06 07 /0 /0 /0 /0 /0 /0 /0 /0 /0 /0 1 1 1 1 1 1 1 1 1 1 Date Date Fig. 8. The figure shows the accuracy of the reputation-based classifier Fig. 9. The figure shows the false positive rate of the reputation-based as determined post-facto from the production deployment of CAMP. classifier as determined post-facto from the production deployment of CAMP. tp fp tnare defined as follows; tpr := tp+fn , fpr := fp+tn , tnr := tn+fp is optimized to minimize false positives. fnand fnr := tp+fn . At the end of the measurement period, the overall accuracy tp+tn The labels to compute the rates are taken from the binary of CAMP was 98.6%. We compute this as tp+tn+fp+fn . Theanalysis system for binaries that could be matched to rep- dominating factor in overall accuracy comes from our trafficutation requests. See Section V-B for an evaluation of the distribution; the majority of requests are for benign binariesaccuracy of the labels. The results are shown in Figure 8. and our system exhibits low false positive rates. As with falseDuring the measurement period, the true negative rate was positive and true negative rates, accuracy increases furthergreater or equal to 98% and the false positive rate was around when taking the client-side whitelists into account.1% for most of the measurement period but for three occasions In June 2012, the true positive rate was around 70% andwhere it increased to around 2%. the false negative rate around 30%. CAMP’s ability to detect A more detailed graph of the false positive rate is shown 70% of recent malware without access to the content of thein Figure 9. We see that the false positive rate was noticeably binary validates the reputation-based detection approach. Welower than 2% for most of the measurement period. When provide a more detailed comparison between CAMP and AVtaking into account that the web browser makes local reputa- engines in Section V-D.tion decisions for about 70% of downloads, the effective true In designing CAMP, our main consideration was avoid-negative rate increases from 98% to 99.5% and the effective ing false positives and as such the binary analysis pipelinefalse positive rate decreases from 2% to 0.6%. While false frequently prefers false negatives over false positives whichpositives are inherent to our content-agnostic approach, our has a negative impact on all measurements presented here,criteria for choosing CAMP’s reputation algorithm parameters e.g. a false negative in the binary analysis pipeline could
  16. 16. 300000 1e+06 ratio/ip/client all malware requests ratio/site/client signed malware requests ratio/site/analysis signed/untrusted malware requests ratio/ip/analysis 250000 binary/unknown Number of client requests with blacklist label multiple rules 100000 200000 Number of Requests 10000 150000 1000 100000 100 50000 0 10 03 04 05 06 07 07 07 07 07 07 07 07 /0 /0 /0 /0 /0 /1 /1 /1 /1 /2 /2 /2 1 1 1 1 1 2 4 6 8 0 2 4 Date Date Fig. 10. The graph shows the stacked contribution to detection in cases Fig. 11. The graph shows the number of client requests corresponding to where a single AND gate or rule is responsible for labeling a download as binaries labeled as malware by the binary classifier. It also shows requests malicious or unknown. It also shows the number of cases in which multiple to malware were the binary was signed. rules flagged the binary download.result in an incorrect false positive for the reputation-based day, 74% no longer appeared in our requests the next day. Thisclassifier. Specifically, when CAMP detects a malware binary percentage increased to 80% over 10 days. We hypothesizecorrectly, but the binary analysis system has a false negative, that this is because such sites are malicious and rotate toour evaluation will count this as a false positive for CAMP. avoid blacklists. This corroborates previous findings that social As discussed in the previous section, each individual AND engineering domains rotate quickly to avoid blacklists [27].gate achieves only low recall. Our hypothesis was that dif- 0.02% of the sites transitioned from unknown to bad over theferent AND gates would combine to increase the overall 10 day period, 8% transitioned to unknown to clean, and theperformance of the classifier. To measure this assumption, we remaining stayed in the unknown state. We manually analyzedplot how often an AND gate is responsible for detection by the top sites that transitioned to clean, and found that almostitself and how often multiple AND gates detect bad content si- all of them were indeed hosting dangerous downloads, andmultaneously. The results are shown in Figure 10 as a stacked the clean classification was a false negative of our dynamicgraph. The graph has two interesting features. The majority analysis framework. Based on these results, we believe thatof detections are due to a single rule, i.e. each individual the unknown verdict is actually a very strong indicator that arule contributes to the overall detection. The other interesting binary is actually malicious.observation is that the detection of unknown binaries accounts D. Comparison to other systemsfor a significant fraction of malicious verdicts. We show inthe case study below how adversaries frequently rotate the To determine if CAMP provides benefits beyond thatdomains from which they are serving malware and thus never provided by other malware detection systems, we conductedbuild up any reputation. The drop in the detection of unknown two different measurements in which we compared reputationbinaries around March 24th is due to a change in data formats decisions from CAMP to results from Anti-Virus engines asthat resulted in dropping the contributions of the unknown rule well as web-based malware services.from the graph. Over a period of two weeks we collected a random sample The rule for labeling binaries as unknown requires that of 10, 000 binaries that were labeled as benign by CAMPa binary is not signed by a trusted signer. To understand as well as approximately 8, 400 binaries that were labeledhow often malicious binaries with trusted signatures occur in as malicious by CAMP. To avoid bias in binary selection,our analysis, we extracted the signature state from binaries for example due to popular download sites which might bewe analyzed from client requests post-facto. As shown in over represented in a completely random sample, we sampledFigure 11, the majority of requests to malware are for unsigned by site rather than by individual binaries. We compared thebinaries. That makes a trusted signature a good predictor for results from CAMP against the results from four different AVthe likelihood that a binary is benign. engines1 that scanned the binaries on the same day as CAMP We also wanted to understand how often a binary that is made its reputation decision. We conducted the AV scans onlabeled as unknown transitions to clean or to malicious after the same day to approximate the performance users mightit has been evaluated by our dynamic analysis pipeline. To see when AV engines scan their downloads. The AV enginesdo so, we analyzed requests from clients post-facto, for a ran in a sandboxed environment, and were not permitted anyperiod of 10 days in November 2012. Of all the sites that 1 Due to licensing restrictions, we cannot disclose the specific AV enginesserved binaries that were classified as unknown on the first we used in our measurements.