Recommended
More Related Content
What's hot
What's hot (20)
Similar to Protecting Apps from Hacks in Kubernetes with NGINX
Similar to Protecting Apps from Hacks in Kubernetes with NGINX (20)
More from NGINX, Inc.
More from NGINX, Inc. (20)
Recently uploaded
Recently uploaded (20)
Protecting Apps from Hacks in Kubernetes with NGINX
- 1. Protecting Apps from Hacks in Kubernetes with NGINX
- 2. ©2022 F5 2 Your Speakers today… Akash Robert
- 3. ©2022 F5 3 Threats to Modern Apps
- 5. ©2022 F5 5 Application Layer Vulnerabilities are still #1 Source: Forrester, The State of Application Security, 2021
- 6. ©2022 F5 6 DDoS Attacks Hit an all-time High in 2022 CONFIDENTIAL - ISMC FY23 This graph shows a Silverline ISP/hosting customer experiencing a DDoS multi-vector attack of a 1.4 Tbps Volumetric Attack and a 100 Mbps Application Attack at the same time.1 1 Attack, multiple vectors DDoS attacks larger than 250 Gbps grew by 1300% in 20213 1,3 F5 Labs: 2022 Application Protection Report: DDoS Atack Trends, March 16, 2022 2Securelist.com: DDoS attacks in Q1 2022, April 25, 2022 DDoS attacks increased 4.5x in Q1 2022 over Q1 20212
- 7. ©2022 F5 7 Applications in Kubernetes Monolithic application running on a virtual machine A Microservices based application in Kubernetes Application layer vulnerability exploit Application layer DDoS attacks
- 8. ©2022 F5 8 Protecting Apps from Hacks and (app) DDoS
- 9. ©2022 F5 9 Mitigating Application Layer Vulnerabilities GET HTTP://example.com?ID=`admin' AND 1=0 UNION ALL SELECT 'admin', '81dc9bdb52d04dc20036dbd8313ed055' Attacking request Response with Payload Assess Act Block Alert Inspect
- 10. ©2022 F5 10 Mitigating Application Layer DDoS Detect Act Baseline
- 12. ©2022 F5 12 NGINX App Protect WAF and DoS
- 13. ©2022 F5 13 No Action Alert Block NGINX App Protect WAF Request Attack Signatures Treat Campaigns HTTP/gRPC Compliance Evasion Techniques Parameter Values Meta Characters File extensions JSON/XML Validation Methods Request Size Level Meaning 0 No threat 1-2 False Positive 3 Needs Investigation 4-5 Threat Response Response Code Response Content Violation Rating Engine
- 14. ©2022 F5 14 App Protect DoS Requests DoS Signatures Bad Actors Automation tools Level Meaning 0 Do Nothing 1 Block Bad IP’s 2 Block Bad Requests 3 Global Rate Limit DoS Attack Detection Engine TLS Fingerprinting Baseline Site Model Service Health Check Detect Anomaly Examine user behavior Create Rules Apply Rules Baseline Model
- 15. ©2022 F5 15 Adaptive DDoS mitigation 3 Global Rate Limit 2 Block Bad Requests 1 Block Bad IP’s Detect Anomaly Site stress monitoring
- 16. ©2022 F5 16 Protecting Apps in Kubernetes
- 17. ©2022 F5 17 Exposing Apps in Kubernetes Small, static deployments Scalable, dynamic deployments
- 18. ©2022 F5 18 NGINX App Protect WAF Deployment Options
- 19. ©2022 F5 19 Applications in Kubernetes (Afurther sweeping generalization) Monolithic application running on a virtual machine A Microservices based application in Kubernetes Deployment Frequency Update/Deployment Methods Infrequent Patched/Manual update Frequent Redeploy/Rolling/Canary
- 20. ©2022 F5 20 NGINX App Protect WAF Deployment Options
- 21. ©2022 F5 21 Fully Integrated Solution • Configured from familiar, powerful K8s API • Integrates security and WAF natively into the CI/CD pipeline Business Benefits • Reduce complexity • Accelerate time to market • Resilience, visibility and security Secure your Kubernetes Apps and Services with NGINX App Protect WAF and NGINX Ingress Controller we are here SECURE YOUR KUBERNETES ENVIRONMENT AT SCALE
- 22. ©2022 F5 22 Using the Ingress API Familiar and standard Reasonably powerful • Monolithic • Can get complicated with annotations/snippets etc. Using NGINX Ingress CRD’s Modular reusable objects (policies, routes etc) Less Complex Similar to new Gateway API • NGINX-only • May still need snippets NGINX Ingress Choices
- 23. ©2022 F5 23 CONFIDENTIAL Host TLS Upstreams Routes - Path Action Split Match Route ErrorPage pass redirect return proxy delegation optional Host TLS Upstreams Routes - Path Action Split Match Route ErrorPage pass redirect return proxy delegation optional NGINX Ingress Resources – Rich Capabilities Host TLS Policies Upstreams Routes - Path Policies Action Split Match Route ErrorPage pass redirect return proxy delegation VirtualServer pass redirect return proxy pass redirect return proxy Host Upstreams Subroutes - Path Policies Action Split Match ErrorPage pass redirect return proxy VirtualServerRoute NGINX server configuration NGINX http configuration Server and HTTP snippets NGINX location configuration Location snippets Policies Access Control Rate Limiting Auth (JWT, OIDC) MTLS (Ingress/Egress) App Protect WAF
- 24. ©2022 F5 24 Configuration apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: juice-shop-vs spec: host: owasp-juiceshop.example.com policies: - name: waf-policy-prime tls: secret: app-secret upstreams: - name: juiceshop service: juiceshop port: 8000 routes: - path: / action: pass: juiceshop apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: juiceshop annotations: appprotect.f5.com/app-protect-enable: "True" spec: ingressClassName: nginx rules: - host: "juiceshop.nginx.marketing" http: paths: - backend: service: name: juiceshop-v2 port: number: 80 path: / pathType: Prefix
Editor's Notes
- Intro the speakers
- This is research on the opinions of on the biggest challenges to running applications in containers, Kubernetes and or serverless (from now on I’ll call these modern apps). As you can see ## security is number 1. Complexity is also a problem, along with the skill to manage it. These are opinion based, so maybe they are more emotional than factual?
- Well no, it looks like application layer security is still a big problem - with web app and software vulnerabilities (which have a lot of overlap) scoring very high on this survey on the causes of compromise. So application layer exploits are a serous threat. I’m not going to talk about the costs of compromise, those figures are quite specific to an organization – but we all know you don’t want to get hacked.
- As if that’s not bad enough, DDoS is a significant ptoblems. What we are seeing in the current Marketplace for 2022, is that DDoS attacks are at an all-time high. Just a quick review that a distributed denial of service or DDoS attack is a cyberattack where multiple connected devices, known as a botnet, are controlled from a central point with the goal of the attacker sending malicious requests to consume all available network bandwidth or resources on a target network, system or website making them slow to respond, or shutting them down entirely and preventing legitimate requests from being fulfilled. Why the increase in DDoS attacks? Several security trade publications including a recent report from Secure List point to the spike in DDoS attacks this year due to the war in the Ukraine and the current geopolitical situation which has resulted in subsequent attacks on businesses specifically in the government and financial sectors. These types of DDoS attacks are being motivated by what is called patriotic hacktivism. The Secure List also notes these trends, that DDoS attacks have grown by 4.5x in volume in Q1 of 2022 than at the same time last year and that DDoS attacks larger than 250 Gbps grew by 1300%. Additionally, according to F5 Labs 2022 report on DDoS Attack Trends based on the analysis of attack data collected by F5s Silverline team from 2021, DDoS attacks are continuing to grow in both size and complexity. In recent years bad actors have increasingly been launching what you see in this graph here, a multi-vector attack, which shows a combined volumetric and application attack at the same time. In November 2021, Silverline observed and mitigated the largest attack it had ever seen. The attack was targeted at an ISP/hosting customer and lasted for just four minutes and reached its maximum attack bandwidth of almost 1.4 Tbps in only 1.5 minutes. Even though the volumetric attack size dwarfs the application attack size, the application attack is just as serious. The resources being requested by the application attack could have easily consumed a web or database server.
- OK, here we are going to compare a traditional application running on a virtual machine, or maybe even a physical server. Is it vulnerable to an application layer hack? Yes. How about a DDoS attack. - well if it’s publicly facing, then yes too. What about a modern app? Built of multiple services running in Kubernetes? Well- honestly there is no difference. Your app needs exactly the same protection – but probably delivered in a different place and managed a different way.
- We’re not going to cover massive volumetric attacks here, but F5 does have a DDoS service with tens of terabits of capacity, globally distributed and ready to protect you: F5 distributed cloud. While massive volumes and huge numbers of connections get the headlines, layer 7 (as we saw earlier) is often part of the attack, and needs different protection tools.
- Let’s talk about defending against application layer compromise. One solution is to write perfect software. Software that always sanitizes user input and isn’t vulnerable to any of the other attacks out there. Oh, and don’t forget to only use perfect software libraries and frameworks in your app too. OK, so that’s not going 0t happen, so what you need is a device that ca inspect (and decrypt) ## your application traffic, then assess ## whether it’s a treat, then take action## Sounds simple, but anyone who has been working with traditional enterprise security products like a Web Application firewall knows that’s not always the case. Or ever the case, in fact.
- OK, now let’s talk about application layer DDoS attacks. This tends to work a bit differently because behavioral detection is the preferred way . So first we need to build a baseline of the site behavior and performance. So that wen an attack occurs ## we can detect a change, and then act## - again this sounds easy, but accurately detecting an attack, and creating a way to detect the specific traffic causing the problem, and then blocking it in a way that still allows good traffic through is far from simple.
- NGINX offers solutions for both application layer attacks and application DDoS – inventively called NGINX app Protect WAF and NGINX App Protec DoS. Between then they can help prevent both application layer DDoS attacks and Application layer exploits like SQL injection (yes still a thing, despite being old enough to drink), or cross site scripting. Lets take a moment to see how they work in a bit more detail.
- First app protect waf – when we talked about the inspect capability a while ago this answers the “inspect for what” question you might have had ## I’m not going to go through all the details here – it’s very well documented, but suffice it to say we look for obvious things like signatures, but also compliance to protocol standards, request sizs, the values of parameters etc. We also look at the repose content to ensure that doesn’t contain thinks like credit cards numbers. Of course, just because a request has triggered a signature, that doesn’t mean that we automatically want to block it, we need to be a bit more careful if we are not going to overwhelm the alerting system, or block traffic we shouldn’t. ## This is where the violation rating engine comes it. The engine takes a number of factors into account to build a model and decide what the rating is, from 0-5. OK, the final step is to take action ## Obviously we can do mothing, alert or block the request – and by default only ratings of 4 or over cause a blocking action, but you can configure your policy how you like to block or alert at different levels. So that’s WAF, what about DoS?
- DoS works in a slightly different way – by building baseline model of load, ## and then noticing na anomaly – art which point the traffic can be examined ## rules created to detect the problematic traffic, then used to take action on the incoming traffic.
- The taking action part is adaptive – we want to do the least invasive thing first, so as to block only the bad traffic – so we will apply the first level of blocking, check the effect on the site load, then if there is no change – aspply a borader rule set, and then of still no change we can start rate limiting all traffic – not ideal, but the best choice if nothing else works,
- So that’s a high-level overview of how you protect applications – what about applications in Kubernetes? First, we should look at how you expose apps to the cold-hard world outside the cluster.
- LoadBalancer with NodePort is one of the most popular ways to expose apps. A LoadBalancer service accepts external traffic and distributes it across the nodes in the cluster. A NodePort service opens a specific port on every node in the cluster, and forwards any traffic sent to the node on that port to the corresponding app. It’s most often used in a cloud platform and is a good choice for small, static deployments. However, using LoadBalancer to expose your apps becomes challenging in dynamic environments where app pods need to scale to meet changing levels of demand. Because each pod gets its own IP address, a popular app can have hundreds – or even thousands – of IP addresses to manage. In most cases, the external load balancer connects to the services via NodePort as shown in the following diagram. While this guarantees traffic is distributed evenly across the nodes, load balancing to the services isn’t possible, so you might encounter service overload. Ingress controllers and gateways provide far more granular control and management of external interactions with pods. You can easily use an Ingress controller not only to control ingress traffic but also to deliver service‑level performance metrics and as part of a security policy. Ingress controllers have many features of traditional external load balancers, like TLS termination, handling multiple domains and namespaces, and of course, load balancing traffic. Ingress controllers can load balance traffic at the per‑request rather than per‑service level, a more useful view of Layer 7 traffic and a far better way to enforce SLAs. Also, many organizations have use cases that benefit from deploying an external load balancer with multiple Ingress controller instances. This is especially common when organizations need to scale Kubernetes. The tools are typically managed by different teams and used for different purposes: Load balancer (or ADC): Owner: A NetOps (or maybe SecOps) team Use case: Outside Kubernetes as the only public‑facing endpoint for services and apps delivered to users outside the cluster. Used as a more generic appliance designed to facilitate security and deliver higher‑level network management. Ingress controller: Owner: A Platform Ops or DevOps team Use case: Inside Kubernetes for fine‑grained load balancing of north‑south traffic (HTTP2, HTTP/HTTPS, SSL/TLS termination, TCP/UDP, WebSocket, gRPC), API gateway functions, and centralized security and identity.) This diagram shows the load balancer handling distribution of the traffic across multiple clusters, while the clusters have Ingress controllers to ensure equal distribution to the services.
- NGINX App rptoect WAF and app protect Dos are the solutions we will be talking btut todat NGINX App Protect F can be deployed in several different ways: 1 – It can live on NGINX Plus when NGINX plus is used as a load balancer, 2 – If you’re using NGINX Plus as an API Gateway, you can install NGINX App Protect WAF in that instance to protect API’s 3 – You can use it as an Ingress controller for Kubernetes clusters 4 – You can use it as a Per-Service proxy for microservices or 5 – You can also use it as a Per-Pod Proxy for microservices Each one covers different scenarios and fulfills specific needs in terms of north-south or east-west based security. With the full ability to have mix- and-match kind of deployments that impose virtually no restrictions on how security can be enforced, it always guarantees full control, auditing, and traceability. For our conversation today, we will focus on using it with the Ingress Controller. architecturally as shown in this diagram. At the edge, at the ingress, as a k8s service and as a sidecar. Our customers can deploy NGINX App Protect WAF in different environments:
- OK, back to my application characteristic matrix – and as the title says, this is a generalization, but I’ve found that it’s generally true. Monolithic apps tend to be deployed less frequently## – where as modern apps are being updated deployed frequently, ## Monolithic apps tend to be upgraded in place, or patched live. ## Modern apps tend to go for rolling redeploys, canary deployments and so on## Why does this matter? Well, I think it gives us an indication of where we should put out app protect services.
- For a lot of use cases, it makes sense to put the protection services as part of ingress ## – because we are already using ingress to direct traffic, it’s already aware of the changes going on with services etc, and if we add a new service, or delete one, then we can add and remove the App protect functions as they are bound to the ingress. It makes operations incidents less likely where either old configuration is hanging around externally, or new services go into place with no protection.
- So the NGINX ingress controller – it’s a really good ingress controller – that you can also add App-Protect as well. You can use the standard Kubernetes ingress API (or chose to use NGINX custom resources – more on that shorty) – it’s made by us, and supported by us too. Also depicted in this diagram is NGINX Service mesh – we’ll probably do a whole separate webinar on that soon, so stay in touch. So now you can have an ingess controller, configured via manifests, that brings powerful security tools right into your k8s cluster.
- I said a minute ago that you could use either the standard ingress API or custom resources – and, hopefully you then thought – ok, but why choose one or the other? <talk through the pro’s and con’s>
- I really like the CRD’s because of the logical structure You can do so with NGINX Kubernetes objects (VirtualServer, VirtualServerRoute, Policy objects) enabling distributed configuration architectures, where administrators can easily delegate configuration to other teams. Teams (Identity, DecSecOps, DevOps) are responsible for policy objects (NGINX App Protect WAF, SSO, ZT, traffic control). And these policies get referenced in many areas of the ingress load balancing configuration (Both at the hostname and path level). At the hostname level, policy objects will be referenced in the VirtualServer Object At the HTTP path level, policy objects will be referenced in the VirtualServerRoute objects Now we are going to see this distributed configuration in action where we implement the WAF self-service model.
- As you can see the configuration is very familiar to anyone working with k8s manifests. Explain config
- OK, we’re just going to demo app protect WAF – because simulating a DDoS attack is difficult and boring to watch!