Developers are employing bots, spiders and scrapers with increasing frequency to gather and utilize information gleaned from websites. Bots and scrapers can be divided into four categories, depending on desirability and aggressiveness. Knowing and understanding the different categories and how to mitigate the risks they pose is an important component of a web security strategy. Learn more about how to evaluate which bots to allow access to your company’s website in this summary presentation, and then download the full report at www.stateoftheinternet.com/security-reports

1. akamai.com
[Q4 2014]

2. • Prevalence of third-party content bots is increasing
• Used to collect data from other websites
• Poorly coded bots can impact site performance
• May resemble a denial of service (DoS) attack
• Secure web presence requires a comprehensive
understanding of how they affect performance
[Download the Q4 2014 Global DDoS Attack Report for
supporting data and analysis]
= bots, spiders & scrapers
2 / [The State of the Internet] / Security (Q4 2014)

3. = purposes of bots and scrapers
3 / [The State of the Internet] / Security (Q4 2014)
Akamai has seen bots and scrapers used for many purposes,
such as:
• Setting up fraudulent sites
• Reuse of consumer price indices
• Analysis of corporate financial statements
• Search and metasearch engines
• Data mashups
• Analysis of stock portfolios
• Competitive intelligence
• Location tracking

4. = example of a bot targeting a financial aggregator
4 / [The State of the Internet] / Security (Q4 2014)
In this example, the bot targeted the financial
aggregator to scrape a large amount of data quickly.
Figure 1: Bot targeting financial aggregator to scrape large amount of data quickly

5. = four categories of bots and scrapers
Figure 2: Desirability is based on how much the site owner wants to host the bot. Aggressiveness the rate of
requests from the bot and its impact on site availability.
5 / [The State of the Internet] / Security (Q4 2014)

6. • Mitigation techniques vary depending on bot classification
• Volume of requests can help determine platform
• The sequence and pages a bot targets can reveal intent
• The user-agent header can sometimes provide a unique and
identifiable user agent
• Additionally, Whois can sometimes identify bot owners
= triage and categorization
6 / [The State of the Internet] / Security (Q4 2014)

7. = corresponding mitigation strategies
Figure 3: Mitigation strategies are based on bot desirability and aggressiveness
7 / [The State of the Internet] / Security (Q4 2014)

8. • Bots and scrapers will continue to affect organizations
in all industries
• Every website should develop a strategy to address
and mitigate undesirable bot behavior
• Each organization will also need to a framework to evaluate
which bots to allow access to it site
• Companies will also need to analyze and modify security
policies to keep them current with the changing times
= moving forward
8 / [The State of the Internet] / Security (Q4 2014)

9. • Download the Q4 2014 State of the Internet Security Report
• The Q4 2014 report covers:
/ Analysis of DDoS attack trends
/ Breakdown of average Gbps/Mbps statistics
/ Year-over-year and quarter-by-quarter analysis
/ Types and frequency of application-layer attacks
/ Types and frequency of infrastructure attacks
/ Trends in attack frequency, size and sources
/ Where and when DDoSers launch attacks
/ Case study and analysis
= Q4 2014 global attack report
9 / [The State of the Internet] / Security (Q4 2014)

10. • StateoftheInternet.com, brought to you by Akamai,
serves as the home for content and information intended to
provide an informed view into online connectivity and
cybersecurity trends as well as related metrics, including
Internet connection speeds, broadband adoption, mobile
usage, outages, and cyber-attacks and threats.
• Visitors to www.stateoftheinternet.com can find current and
archived versions of Akamai’s State of the Internet
(Connectivity and Security) reports, the company’s data
visualizations, and other resources designed to put context
around the ever-changing Internet landscape.
= about stateoftheinternet.com
10 / [The State of the Internet] / Security (Q4 2014)