Successfully reported this slideshow.
Your SlideShare is downloading. ×

Api design best practices from a hacker's view

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
Hotel api integration
Hotel api integration
Loading in …3
×

Check these out next

1 of 40 Ad

Api design best practices from a hacker's view

Download to read offline

In this ppt I will talk about how I hack freelancer.com and another travel web site to get here API so I can crawl information from them.

In this ppt I will talk about how I hack freelancer.com and another travel web site to get here API so I can crawl information from them.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Advertisement

Similar to Api design best practices from a hacker's view (20)

Advertisement

Recently uploaded (20)

Api design best practices from a hacker's view

  1. 1. API DESIGN BEST PRACTICES FROM A HACKER’S VIEW
  2. 2. • Overview • Stories • Crawl all projects and bids from Freelancer.com • Crawl 6 billion flight ticket price from a travel web site • Summary
  3. 3. MONOLITHIC APP • Hide system information inside app • No internal sys call is exposed to outside
  4. 4. MICROSERVICE APP • Hackers know your system better • Service calls are exposed to user • RESTful API as standard,easyto guess • Need to consider security between every service
  5. 5. TRIDITIONAL MODERN XPATH WEB PAGE API Pure Data
  6. 6. STORY 1 CRAWL FREELANCER.COM
  7. 7. FREELANCER.COM 8M Project Information Bid Information
  8. 8. • Reputation and price, which is the most important factor for a success bid? • How canI get most chance to be awarded when bidding for Australia employer? • Should I put a lowest price or should I do more project to earn reputation
  9. 9. HOW CAN I GET THE INFORMATION AS FAST AS POSSIBLE?
  10. 10. https://www.freelancer.com/projects/Javascript/Web-Page-Scraper/
  11. 11. • Need a HTML parser and javascript executor • Heavy work for both cpu and bandwidth • Not easy to iterate through all the projects
  12. 12. TIP: MOST OF THE TIME MOBILE SITE IS MUCH EASIER TO GET INFORMATION
  13. 13. https://m.freelancer.com/projects/Javascript/Web-Page-Scraper/#info
  14. 14. RESTFUL APIS
  15. 15. https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page- Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upg rade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&us er_status=true https://www.freelancer.com/api/projects/0.1/projects/9844976/bids/?compact=true&limit=20&offset=0&reputatio n=true&user_avatar=true&user_details=true https://www.freelancer.com/api/projects/0.1/projects/9844976/?compact=true&full_description=true&job_details= true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_ employer_reputation=true&user_reputation=true&user_status=true https://www.freelancer.com/api/projects/0.1/projects/${id}/?compact=true&full_description=true&job_details=tr ue&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_ employer_reputation=true&user_reputation=true&user_status=true
  16. 16. API Rate Limit 1000 / HOUR 8M / 1k = 8k HOUR = 333 DAYS
  17. 17. 172.246.149.100 216.219.130.171 95.227.99.197 17.124.253.149 137.238.189.207 34.155.214.35 • Number of threads depends on how many proxies you have • Https proxy is hard to find • Proxies are unstable • Proxies will be used out quickly • High cost if you buy proxies WORKAROUND : USE HIGH ANONYMOUS PROXY
  18. 18. 160.124.89.71 13.193.36.236 182.3.152.44 85.72.136.122 …… • Loads of IPs, canbe changed every 10s • High quality socks proxies across the world • Able to use docker to start 10 tor clients in 1 minute WORKAROUND: USE TOR NETWORK
  19. 19. USING THESE HACKS I MANAGED TO GET ALL THE PROJECTS AND BIDS IN 10 DAYS USE A SINGLE DIGITALOCEAN 5$ SERVER
  20. 20. WHAT DO I LEARN?
  21. 21. API Rate Limitation Mobile API
  22. 22. Easy to guess filters Predicable URL https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page- Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true& upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=tr ue&user_status=true Information leak
  23. 23. HOW CAN WE FIX THEM?
  24. 24. ONLY SUPPLY INFORMATION CLIENT NEEDS
  25. 25. MAKE SURE URL IS NOT PREDICTABLE https://www.freelancer.com/api/projects/0.1/projects/UUID/?compact=true&full_description=true&job_details=tr ue&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_ employer_reputation=true&user_reputation=true&user_status=true
  26. 26. REDUCE ANONYMOUS NETWORK ATTACK • If your customers are in AU only, restrict access when IP address is outside AU • Set different limitation based on location • 1k/h API usage • 100/h API usage • Captcha to verify human
  27. 27. STORY 2 LEARN FROM CAWLING FLIGHT TICKET PRICE
  28. 28. How many days ahead do I need to get a cheapest price? I need to crawl as many flight ticket price and analysis.
  29. 29. FIND API FROM MOBILE PAGE
  30. 30. data=%7B%22searchType%22%3A%…… useNative=true&ttid=201300@travel_h5_3.1.0&appKey=12574478 t=1426062775998&sign=3feb52aed67967a2c47aa7a2b9f2a417 If you access the same url to reproduce API calls, it will after 10 seconds ANALYSE API • Parameters inside data parameter: • Fixed parameter: • Sign
  31. 31. HOW CAN WE GENRATE A VALID API CALL?
  32. 32. FIND TRIGGER POINT Search source code to find API endpoint
  33. 33. REFORMAT SOURCE CODE • Reformat code to get readable source code • Help to set breakpoint
  34. 34. FIND API URL GENERATOR • Trace down the code to find out howto generate the url
  35. 35. FIND OUT TOKEN GENERATION ALGORITHM • Set breakpoint and watch variables to find out the secret
  36. 36. WHAT DO I LEARN?
  37. 37. • Use time token to generate dynamic urls • Use parameter sign token to verify parameter • Prevent repeat API calls • JS obfuscated code is easy to hack
  38. 38. SUMMARY
  39. 39. • Make sure url is not predictable • Only supply information client needs • Reduce anonymous network attack • By different strategy to different location • Use time token to generate dynamic url • Use sign to verify request is valid
  40. 40. THANK YOU Github: derekhe

×