Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Application DoS In Microservice Architectures
Similar to Application DoS In Microservice Architectures (20)
Recently uploaded
Recently uploaded (20)
Application DoS In Microservice Architectures
- 1. Starting the Avalanche: Application DoS In Microservice Architectures Scott Behrens Jeremy Heffner
- 2. $1.71
- 3. Photo of a battering ram by Flickr user Patrick Denker; License: https://creativecommons.org/licenses/by/2.0/
- 4. Jami al-Tawarikh (Compendium of Chronicles) by Rashid ad-Din. Edinburgh University Library via Wikipedia Commons.
- 5. Outline What is Application DDoS (3 minutes) Intro to Microservices (4 minutes) Application DDoS in Microservices (7 mins) Framework Discussion (9 minutes) Case Study/Introduction to Repulsive Grizzly and Cloudy Kraken (15 minutes) Demo (2 minutes) Mitigation Strategies (3 minutes) Future Work/Call to Action (2 minutes)
- 6. DDoS focused on application layer logic
- 7. How Novel is Application DDoS? Akamai’s State of the Internet Security Report." https://www.akamai.com. N.p., 19 Feb. 2017. Web.
- 8. Microservices are a collection of small loosely coupled but collaborative services
- 10. Microservice Primer: API Gateways and Client Libraries Interface for middle tier services Services provide client libraries to API Gateway Diagrams provided with permission by microservices.io
- 11. Microservice Primer: Circuit Breaker Helps with handling service failures How do you know what timeout to choose? How long should the breaker be triggered? Diagrams provided with permission by microservices.io
- 12. Microservice Primer: Cache Speeds up response time Reduces load on services fronted by cache Reduces the number of servers needed to handle requests https://github.com/netflix/evcache
- 14. Less Common But Interesting Application DDoS Queueing Client Library Timeouts Healthchecks Connection Pool Things That Don’t Autoscale
- 15. How Logical Work Per Request Differs Most Monolithic Application DDoS Often 1 to 1 Some Microservice Application DDoS Often 1 to Many
- 16. PROXY PROXY PROXIES API GATEWAY WEBSITE Middle Tier Service Middle Tier Service Middle Tier Service Middle Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Edge Middle Backend New School Microservice API DDoS > python grizzly.py …
- 17. PROXY PROXY PROXIES API GATEWAY WEBSITE Middle Tier Service Middle Tier Service Middle Tier Service Middle Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Edge Middle Backend New School Microservice API DDoS > python grizzly.py … Fallback or Site Error API Gateway making many client requests Middle tier services making many calls to backend services Backend service queues filling up with expensive requests Client Timeouts, circuit breakers triggered, fallback experience triggered
- 18. API GATEWAY Middle Tier Service Middle Tier Service Middle Tier Service Middle Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Edge Middle Backend Microservice API DDoS - Simplified Single request asking for 1,000 objects not in cache
- 19. Single Request Asking for 1,000 Objects not in Cache*
- 20. Cache Miss Attack Figure out what is in the cache Make calls that require lookups outside of cache
- 21. API GATEWAY Middle Tier Service Middle Tier Service Middle Tier Service Middle Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Edge Middle Backend Microservice API DDoS - Simplified Middle tier has to call 3 backend services, or 6,000 calls. Single request asking for 1,000 objects not in cache Middle tier client library doesn’t support batching Results in 2 RPC calls per object, or 2,000 RPC calls
- 22. Workflow for Identifying Application DDoS - Part 1 Identify the most latent service calls Investigate if latent calls allow for manipulation Determine API Gateway error conditions, thresholds, and library timeouts Tune payload to fly under WAF/Rate Limiting Test hypothesis Scale your test using Cloudy Kraken (orchestrator) and Repulsive Grizzly (attack framework)
- 23. Identifying Latent Service Calls
- 24. Example Workflow - Ordinary API Call POST data seems to be encrypted. Modifying POST body seems to return fast error code Changing I/O didn’t result in increased latency of API calls
- 25. Example Workflow - Interesting API Call Adding more items to list results in longer response time Adding too many items results in a special error code (exp. 502) Changing I/O resulted in increased latency of API calls
- 26. Identifying Latent Service Calls
- 27. Microservice Application DDoS: Attack Patterns Range Object Out per Object in Request Size Combination
- 28. Application DDoS Technique: Range
- 29. Application DDoS Technique: Object Out Per Object In
- 30. Application DDoS Technique: Request Size
- 31. Application DDoS Technique: Combination <--What about N languages? <--What about more object fields?
- 32. Build a List of Indicators on API Health Healthy Response API Gateway Timeout Client Library Timeout WAF Rate Limiter Framework Exceptions
- 33. Other Indicators HTTP 200 + Latency Empty Response Look for Correlations
- 34. Rate Limited* Service May Scale Up/ Service Healthy Logical Work Per Request # Req Service Healthy Service Impact ServiceImpact
- 35. New School Application DDoS Attack: Case Study
- 36. Making the Call More Expensive
- 37. Workflow for Identifying Application DDoS - Part 2 Identify the most latent service calls Investigate if latent calls allow for manipulation Determine API Gateway error conditions, thresholds, and library timeouts Tune payload to fly under WAF/Rate Limiting Test hypothesis Scale your test using Cloudy Kraken (orchestrator) and Repulsive Grizzly (attack framework)
- 38. Repulsive Grizzly Skunkworks* application DDoS framework Written in Python3 Eventlet for high concurrency Uses AWS SNS for logging analysis Easily configurable
- 39. Repulsive Grizzly: Bypass WAF with Sessions http://nerdprint.com/wp-content/uploads/2014/09/Dumle-Mountain-Cookies-Advertising2.png
- 40. Repulsive Grizzly: Single Node
- 41. Cloudy Kraken
- 42. Cloudy Kraken - What is it?
- 43. Cloudy Kraken - Key Features Fresh global fleet of test instances each run Lots of fresh global IP addresses Easily push new attack code and configs Coordinates the attack timing Safety kill-switch
- 45. Update Code Push the configuration file and attack scripts to S3. Reset the DynamoDB state. Build Environment Configure the VPCs in each region Start up Attack Nodes Launch instances Collect Data Wait for data to come through SNS Tear-Down Tear down and reset the environment in each region Cloudy Kraken Workflow
- 46. We scaled up, time to run the test! Tested against prod Multi-region and multi-agent Conducted two 5 minute attacks Monitored for success
- 48. $1.71
- 49. So What Failed? Expensive API calls could be invoked with non-member cookies Expensive traffic resulted in many RPCs per request (1:7200) WAF/Rate Limiter was unable to monitor middle tier RPCs
- 50. API GATEWAY Middle Tier Service Middle Tier Service Middle Tier Service Middle Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Backend Tier Service Edge Middle Backend Case Study API DDoS 5. Queue continues to fill up, takes longer to return objects 6. Middle Tier services returning 5XX and 200 status codes 1. Agents cycling through multiple session cookies and IP addresses 7. API gateway starts to trigger breakers, returning some 403’s, 503’s, 200’s, 504’s. 3. Objects not in Cache, Cache Miss 4. About 15 seconds to return huge object store8. Automatic scaling of services in progress, but not fast enough 9. CPU Spiking in API Gateway, Gateway Timeouts occurring 2. Each request asking for 7,200 expensive calculations from multiple backends
- 51. Demo
- 52. Microservice Application DDoS: Mitigations Understand which microservices impact customer experience
- 53. Microservice Application DDoS: Mitigations Limit batch and object size in both client and server
- 54. Microservice Application DDoS: Mitigations Rate limiter (WAF) should monitor middle tier signals or cost of request*
- 55. Microservice Application DDoS: Mitigations Rate limiter (WAF) should monitor volume of cache misses*
- 56. Microservice Application DDoS: Mitigations Prioritize authenticated traffic over unauthenticated
- 57. Microservice Application DDoS: Mitigations Configure reasonable client library timeouts
- 58. Microservice Application DDoS: Mitigations Trigger fallback experiences when cache or lookups fail
- 59. Future Work Automated identification of potential vulnerable endpoints Autotuning attack parameters during exercise based up on error condition criteria Testing common open source microservice frameworks/libraries/gateways
- 60. Thanks! https://github.com/netflix-skunkworks/repulsive-grizzly https://github.com/netflix-skunkworks/cloudy-kraken Questions? Come find us in the chill room after the session! Scott Behrens Jeremy Heffner @helloarbit
- 61. Extra Slides
- 62. Repulsive Grizzly: Payload and Header Files Provide payloads in any format you want Headers are provided as a JSON key/value hash Use $$AUTH$$ placeholder to tell grizzly where to place tokens
- 63. Microservice Primer: Architecture Scale Service independence Fault isolation Eliminates stack debt Distributed system complexity Deployment complexity Cascading service failures if things aren’t set up right GOOD BAD