API scraping involves automated extraction of content from APIs for purposes outside of their intended use. While not necessarily malicious, it can harm API owners through lost revenue and intellectual property. To recognize scraping, API owners monitor for abnormal usage patterns and set limits on requests per user. Methods to prevent scraping include rate limits, firewall rules blocking suspicious traffic, IP databases, and machine learning models to detect bots. However, completely blocking scraping may not be possible due to the dynamic nature of the problem.

1. API Scrapping.
How to protect your API against something that is not
necessarily an attack
Artem Demchenkov
Illustration © Caterina Carraro/Billie

2. 2
Artem Demchenkov
● CTO & Co-Founder at Billie
● ardemchenkov@gmail.com
● https://www.linkedin.com/in/artem-demchenkov-76b69934/

3. Billie
We are an innovative financial platform, focused on working capital financing.
Some numbers:
● Start: 01.12.2016
● Closed Beta: 01.05.2017
● Public Launch: 01.06.2017
● Team: 100+
● Engineering Crew: 35

4. Agenda
4
● API scraping may not look like an attack
● Why should we worry then?
● KYCB
● How to recognize that you’re being scrapped
● How to stop API scraping. It is even possible?
● Bonus

5. 5
Let’s begin with numbers

6. Some numbers
6
● 46% of web traffic are scraping bots (Distil Networks, 2016)
● 2% of online revenue is lost due to web scraping (Distil Networks, 2016)
● Global e-commerce sales are 3.53 trillion dollars (Statista, 2019)
● 2% of it is >70 billion dollars

7. 7
API Attacks

8. API Attacks
8
● DDoS
● Injections
● Data Exposure
● Authentication Hijacking
● “Man in the Middle”
● Unencrypted Communication
● Application Abuse
● Parameter Tampering
}Unexpected API usage

9. 9
API Scraping is different

10. API Scraping
10
● Is a pretty traditional and expected user behaviour
● It doesn’t try to hijack or break anything
● Content - is the main point of interest
● That’s why it is not always easy to identify that you are being scraped

11. Traditional communication flow
11
USER UI API DATA SOURCE

12. Traditional communication flow
12
USER UI API DATA SOURCE

13. Scraping communication flow
13
USER UI API DATA SOURCE

14. 14
Scraping is not allowed

15. Google Maps Terms of Service
15
3.2.3 Restrictions Against Misusing the Services.
(a) No Scraping. Customer will not export, extract, or otherwise scrape Google Maps
Content for use outside the Services. For example, Customer will not: (i) pre-fetch, index,
store, reshare, or rehost Google Maps Content outside the services; (ii) bulk download
Google Maps tiles, Street View images, geocodes, directions, distance matrix results, roads
information, places information, elevation values, and time zone details; (iii) copy and save
business names, addresses, or user reviews; or (iv) use Google Maps Content with
text-to-speech services.

16. 16
But not everyone is Google, isn’t it?

17. Scraping communication flow
17
USER UI API DATA SOURCEBOT

18. Scraping communication flow
18
USER UI API EXTERNAL APIBOT

19. What makes API Scraping dangerous for API Owners
19
● Unpredictable expenses
● Income losses
● Loss of intellectual property ownership
● Loss of competitive advantage
● Risk of being reverse engineered

20. 20
KYC
Know Your Customers
KYCB
Know Your Customers’ Behavior

21. How to recognize that your API is being scrapped?
21
Logging Thresholds Monitoring alerts

22. How to set up a proper Data Monitoring
Smart Data-driven Alerts with Prometheus and Grafana
https://youtu.be/GbwyF6xZwwc

23. How to prevent API Scraping
23
HTTP-Request limits Per user, Per time period
Basic Firewall rules Headers, Content-Size, IPs
More extended rules Geography, Pattern Detection
IP databases https://www.abuseipdb.com/
Real-time ML based bot detection Specific services (e.g. Cloudflare)
Data thresholds Num or registrations, num of API calls...

24. 24
They say:
no one can block API scraping completely

25. 25
We say:
it is a race and one who stops first loses

26. 26
Bonus Track

27. Scraping communication flow
27
USER UI API DATA SOURCEBOTHUMAN
This one leaves traces

28. 28
Prior to building a bot,
a human needs to explore your API

29. Thank you
● ardemchenkov@gmail.com
● https://www.linkedin.com/in/artem-demchenkov-76b69934/

30. Check it out. There’s interesting
More in our “Engineering Corner” on Medium
30
https://medium.com/billie-finanzratgeber