This document discusses how advanced persistent bots pose a threat to real estate portals by scraping listing data and spamming contact forms. Traditional homegrown solutions are ineffective against sophisticated bots. Lamudi, a global property portal, was experiencing performance issues and data scraping due to bot traffic 15x higher than human traffic. Lamudi implemented Distil's bot detection and mitigation solution, which uses techniques like device fingerprinting and browser validation to accurately identify and block bots without impacting users. This resulted in the elimination of data scraping, reduced form spam, improved performance, and saved engineering resources.

1. Securing Property Portals
Lamudi Case Study

2. Platforms
Portals
Brokerages
MLS Customers
Premium Brands
Distil in Real Estate and Premium Brands

3. The New Threat Landscape of APBs
Advanced Persistent Bots (APBs)...
Advanced
Mimick human behavior
Load JavaScript
Load external resources
Support cookies
Browser automation (Selenium, PhantomJS)
Persistent
Dynamic IP rotation
Distribute attacks across IP addresses
Hide behind anonymous and peer-to-peer proxies 2015 Distil Bad Bot Report

4. Homegrown Solutions Are Ineffective
Creates a poor user experience Bots appear human in logs Defeated by distributed IP attacks
Defeated by advanced bots Labor intensive Defeated by low and slow crawlers
Defeated by CAPTCHA farms Distributed attacks hard to pinpoint Defeated by peer-to-peer / proxies
Reduces conversions by up to 27% Reactive in nature Reactive in nature

5. Web App Security Requires Complementary Solutions
l
DDoS Mitigation Firewall WAF Distil Bot Protection
Core
Competency
Volumetric attacks
on infrastructure
Network layer attacks Application coding exploits
Automated abuse, misuse, and attacks
(scraping, fraud, account takeover, etc.)
Techniques Scrubbing centers,
Large pipes
Access Control Lists
(ACLs),
Rules-Based
App layer understanding,
ACLs, Rules-Based
Real-time Analysis, Fingerprinting,
Honeypotting, Machine learning,
Behavioral modeling

6. Survey Respondents
100 real estate executives representing
over 600,000 realtors
14 real estate portal operators running
400,000 real estate websites
2015 Real Estate Web Scraping Survey

7. ○ 50% - 75% of bot traffic is from Consumer ISPs
○ 7 of top 10 sources of bad bots are Consumer ISPs
○ Most Consumer ISPs had 1,500+ IPs with bots
Highlights of Bot Sources on Real Estate Websites
The Facts on Scraping Real Estate Data
Top 7 Consumer ISPs
with Bot Traffic
1 Comcast
2 Time Warner Cable
3 Verizon FIOS
4 Charter
5 Cox
6 CenturyLink
7 AT&T Uverse
Highlights of Bot Sophistication
○ 18-45% Automated browsers - mimicking humans
○ 14-25% Already in bot database - fingerprinted, known bots
○ 16-42% Slow crawlers - recycling IPs and user agents

8. About Lamudi
30+ Countries
900,000+ Listings
660+ Employees
Property portal focused exclusively
on emerging markets

9. Lamudi Bad Bot Challenges
Bad Bot Challenges
Bad guys scraped listing data to duplicate
listings, impact SEO, and compete w/Lamudi
Bots are spamming listing agent/owner contact
forms & reducing agent retention & satisfaction
15,000 bad bot requests per minute (15x
human traffic) caused slowdowns
WAF-based IP blocking system used
enginering time and was ineffective

10. Lamudi Selection Criteria
Bot Detection and Mitigation Solution Requirements
Support a complex deployment across several AWS instances with Akamai
Block web scrapers and spammers without impacting human visitors
Accurately identify good bots vs. bad bots
Increase website availability and speed
Detect automated browsing tools
Simple setup for 30+ domains
Little or no maintenance, “self-optimizing” solution

11. Lamudi Results with Distil
Results with ROI
No more scraping data → unique listings = better SEO
No more form spam to agents → higher value leads = $$
Less time addressing agent complaints → Rev. Retention = $$
Increased website performance → Faster site = better SEO
Save 100 engineering hours/mo. → More resources! Save $$
“Distil is the best anti-bot and anti-scraper protection solution
available, hands down.” Oliver Fiege, CTO, Lamudi

12. How the Distil Bot Detection Solution Works
As web traffic passes through Distil, the system
1. Fingerprints each incoming connection and
compares it to our Known Violators Database
1. If it’s a new fingerprint, validates the browser
to determine if it’s a Bot or Not
1. “No Silver Bullet” - Distil randomizes a battery
of challenges to find bots and remain spoof-
proof from the bot coders
1. Based on your settings, Distil automatically
tags, challenges, or blocks the bot

13. Sticky Bot Tracking With No Impact On Real Users
Device Fingerprinting
Fingerprints stick to the bot even if it
attempts to reconnect from random IP
addresses or hide behind an anonymous
proxy or peer-to-peer network
Tracks distributed attacks that would
normally fly under the radar
Without Distil With Distil
Without Impacting Users Sharing the Same IP
Avoids blocking residential users or organizations
that might share the same NAT as the bot or botnet

14. Browser Validation
Detects all known browser automation tools, such as Selenium and Phantom JS
Protects against browser spoofing by validating each incoming request as self
reported
Advanced Bot Detection Increases Accuracy
Behavioral Modeling and Machine Learning
Machine-learning algorithms pinpoint behavioral anomalies specific to your
site’s unique traffic patterns
Self optimizing algorithms improve bot detection and mitigation without
manual configuration

15. Awards and Analyst Recognition
“Analyzing behavior provides the best
chance of detecting and blocking bot-
driven attacks.”
5 Stars across the board.“
Verdict: For monitoring the impact of bots on
a network this is the tool one needs.”
The only anti-bot solution to be included
in Gartner’s Online Fraud Detection
Market Guide
Ovum puts Distil Networks On The Radar.
“Clear innovation compared to similar
services.”

16. www.distilnetworks.com
QUESTIONS….COMMENTS?
C H A R L I E @ D I S T I L N E T W O R K S . C O M
1.703.962.1614
OR CALL CHARLIE ON