Automating Security
Response with
Serverless
Michael Ducy, Red Hat
@mfdii
About me.
Michael Ducy
Cloud Native Security & Transformation, Red Hat
Past: Sysdig, Chef, Orbitz WW
DevOpsDays Columbus Organizer
Cloud Native Security Day Organizer
https://events.linuxfoundation.org/cloud-native-security-day/
@mfdii @mfdii
@mfdii
• Serverless Overview
• Security Response with Serverless
• MLGuard
• Falco and Security Playbooks
• Take Aways
Agenda
@mfdii
• Faster time to production.
• No servers to manage.
• Fine grained metering.
• Continuous scaling.
• Easily integrate with cloud infrastructure.
Serverless Benefits
@mfdii
Event Driven:
• Triggers a function based on an event
• Typically message queue, pub/sub, http webhook
Periodic Execution:
• Executes a function based on time (Hourly, Daily, etc)
Serverless Design Patterns
@mfdii
Open Source Serverless
@mfdii
Knative
Kubernetes Native Serverless
• Support for any language in a
container.
• Heavy focus on deployment
versioning, blue/green deployments,
scaling.
• Triggers: Kafka, Google PubSub,
Amazon PubSub, CronJob, GitHub,
Kubernetes, Camel, Containers.
7
@mfdii
Kubeless: Simple FaaS
Simple Functions as a Service
(Serverless) for Kubernetes:
• Support for Python, Node.js, Ruby, PHP,
Golang, .NET, Ballerina and custom
runtimes
• Event triggers using Kafka, NATS, and
HTTP events
• Prometheus monitoring of functions calls
and function latency by default
• https://kubeless.io/
8
@mfdii
• NodeJS, Go, Java, Scala,
PHP, Python, Ruby, Swift,
Ballerina, .NET and Rust.
• Triggers limited: Cloudant,
GitHub
Apache OpenWhisk
@mfdii
Open Source Serverless
Security
Response
Patterns
MLGuard
@mfdii
MLGuard
• Overcomes limitations of Web Application Firewalls
• WAFs are static and easy to evade: Static & Signature Based
• MLGuard provides anomaly detection
• Serverless-based machine learning system to detect
malicious web requests.
• Developed by Frame.io
https://www.youtube.com/watch?v=T7C0qtA8Mww
@mfdii
Profile an IP based on kinds of http request that it sends
MLGuard Concept
192.168.xxx.xx
{
2xx: 49
3xx: 0
4xx: 1
5xx: 0
}
192.169.xxx.xx
{
2xx: 4
3xx: 2
4xx: 35
5xx: 0
}
Hmm, that’s good That’s Suspicious
@mfdii
MLGuard
Collects logs
Builds model
Deploys
model
Prediction
Notifies
anomalies
All these stages use serverless technologies
@mfdii
MLGuard: Build - Train - Deploy
@mfdii
MLGuard: Build - Train - Deploy
@mfdii
MLGuard: Notifications
@mfdii
MLGuard at Frame.io
Falco & Serverless
Security Response
@mfdii
A behavioral activity monitor
• Detects suspicious activity defined by a set of rules
• Uses sysdig’s flexible and powerful filtering expressions
With full support for containers/orchestration
• Utilizes sysdig’s container & orchestrator support
And flexible notification methods
• Alert to files, standard output, syslog, programs
Open Source
• Anyone can contribute rules or improvements
Falco.
@mfdii
• Containers are isolated processes.
• Processes are scoped as to what’s
expected.
• Container images are immutable,
runtime environments often aren’t.
• How do you detect abnormal behavior?
Anomaly detection.
Architecture.
@mfdii
CUSTOM APP
CONTAINER
NGINX
CONTAINER
FALCO
CONTAINER
Kernel Instrumentation
HOST / OS KERNEL
SYSTEM
CALLS
Instrumentation.
@falco_org
Falco Architecture
FILTER
EXPRESSION
SYSDIG OSS
LIBRARIES
FALCO_PROBE
KERNEL MODULE
or eBPF PROBE
ALERTING
FALCO RULES
WEB
SERVER
USER
KERNEL
EVENTS
SUSPICIOUS
EVENTS
SHELL
STDOUT
FILE
SYSLOG
HTTPs
gRPC
KUBERNETES
AUDIT LOGS &
METADATA
FALCO
client-go
FALCO
SIDEKICK
Falco Rules
@mfdii
- list: bin_dirs
items: [/bin, /sbin, /usr/bin, /usr/sbin]
- macro: bin_dir
condition: fd.directory in (bin_dirs)
- rule: write_binary_dir
desc: an attempt to write to any file below a set of binary directories
condition: bin_dir and evt.dir = < and open_write and not
package_mgmt_procs
output: "File below a known binary directory opened for writing
(user=%user.name command=%proc.cmdline file=%fd.name)"
priority: WARNING
yaml file containing Macros, Lists, and Rules
Falco rules.
@mfdii
fd - File Descriptors
process - Processes
evt - System events
user - Users
group - Groups
syslog - Syslog messages
Based on “Field Classes”. Supported classes include:
container - Container metadata
fdlist - FD poll events
k8s - Kubernetes metadata
ka - Kubernetes Audit Logs
mesos - Mesos metadata
Conditions and Sysdig Filter
Expressions.
@mfdii
Quick examples.
evt.type = setns and not proc.name in (docker, sysdig)
fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin) and write
container.id != host and proc.name = bash
evt.type = open and fd.name = /dev/video0 and not
proc.name in (skype, webex)
(evt.type = create or evt.arg.flags contains O_CREAT)
and proc.name != blkid and fd.directory = /dev and
fd.name != /dev/null
A shell is run in a container
Overwrite system binaries
Container namespace change
Non-device files written in /dev
Process tries to access camera
@mfdii
Sending alerts
• Events matching filter
expression result in alerts
• Rule’s output field used to
format event into alert
message
• Falco configuration used to
control where alert message
is sent
Any combination of..
• GRPC
• Syslog
• File
• Standard Output
• Shell (e.g. mail -s "Falco
Notification"
someone@example.com)
Alerts + outputs.
@mfdii
- rule: Node Container Runs Node
desc: Detect a process that’s not node started in a Node container.
condition: evt.type=execve and container.image startswith node and proc.name!=node
output: Node container started other process (user=%user.name
command=%proc.cmdline %container.info)
priority: INFO
tags: [container, apps]
A custom Falco rule.
Something is
executing a program
In a container based
on the Node image
And the process
name isn’t node
@mfdii
- macro: contains_private_credentials
condition: >
(ka.req.configmap.obj contains "aws_access_key_id" or
ka.req.configmap.obj contains "aws_s3_access_key_id" or
ka.req.configmap.obj contains “password")
- macro: configmap
condition: ka.target.resource=configmaps
- macro: modify
condition: (ka.verb in (create,update,patch))
- rule: Create/Modify Configmap With Private Credentials
desc: Detect creating/modifying a configmap containing a private credential
(aws key, password, etc.)
condition: configmap and modify and contains_private_credentials
output: K8s configmap with private credential (user=%ka.user.name
verb=%ka.verb name=%ka.req.configmap.name
configmap=%ka.req.configmap.name config=%ka.req.configmap.obj)
priority: WARNING
source: k8s_audit
tags: [k8s]
K8s audit log rule example.
Installing and
Integrations
@mfdii
Installing Falco.
• Debian Package
• apt-get -y install falco
• Redhat Package
• yum -y install falco
• Installation Script
• curl -s s3.amazonaws.com/download.draios.com/stable/install-falco | sudo bash
• Docker container
• docker pull sysdig/falco
• Full instructions
• github.com/draios/falco/wiki/How-to-Install-Falco-for-Linux
@mfdii
• Use Helm
• $ helm install --name sysdig-falco-1 stable/falco
• https://sysdig.com/blog/falco-helm-chart/
• Install Falco as Kubernetes Daemonset
• https://github.com/draios/falco/tree/dev/examples/k8s-using-daemonset
• Configuration stored in Kubernetes ConfigMaps
• Conditions in a Falco Rule can leverage Kubernetes metadata to trigger
events
• Falco events can include Kubernetes metadata to give notification context:
• name, id, labels for Pods, ReplicationController, Service, Namespace,
ReplicaSet, and Deployment
Installing Falco on kubernetes.
How can you
use Falco?
@mfdii
• Detect abnormal events with Falco
• Publish alerts to Pub/Sub service (NATS.io)
• Subscribers can subscribe to various FALCO topics to receive alerts:
• FALCO.* - All alerts
• FALCO.Notice - Alerts of priority “Notice” only
• FALCO.Critical - Alerts of priority “Critical” only
• Subscribers can take action on alerts:
• Kill offending Pod
• Taint Nodes to prevent scheduling
• Isolate Pod with Networking Policy
• Send notification via Slack
Response engine &
security playbooks.
@mfdii
Image Placeholder
Insert image and use
this box for sizing
Image Placeholder
Insert image and use
this box for sizing
Image Placeholder
Insert image and use
this box for sizing
Detects abnormal event, Publishes
alert to NATS
Subscribers receive Falco Alert
through NATS Server
Kubeless receives Falco Alert, firing a
function to delete the offending
Kubernetes Pod
https://sysdig.com/blog/oss-container-security-runtime/
Response engine &
security playbooks.
@mfdii
Response engine &
security playbooks.
https://sysdig.com/blog/container-security-orchestration-falco-splunk-phantom/
PUBLISH TO TOPIC
FALCO-NATS
SIDECAR
LINUX PIPE
FALCO CONTAINER
FALCO DAEMONSET
EVENTS
K8S
METADATA
KUBERNETES NODES
kubelet API
APPLICATION DEPLOYMENTS
EXECUTE REACTION
i.e. kill the offending pod
F(x) F(x) F(x)
WEBHOOK
NOTIFICATION
SUBSCRIBE TO
1..N TOPICS
kubernetes
@mfdii
Response engine &
security playbooks.
PUBLISH TO TOPIC
FALCO-NATS
SIDECAR
LINUX PIPE
FALCO CONTAINER
FALCO DAEMONSET
EVENTS
K8S
METADATA
KUBERNETES NODES
kubelet API
APPLICATION DEPLOYMENTS
EXECUTE REACTION
i.e. kill the offending pod
F(x) F(x) F(x)
WEBHOOK
NOTIFICATION
SUBSCRIBE TO
1..N TOPICS
AWS LAMBDA
https://aws.amazon.com/blogs/opensource/securing-amazon-eks-lambda-falco/
@mfdii
Image Placeholder
Insert image and use
this box for sizing
Image Placeholder
Insert image and use
this box for sizing
Image Placeholder
Insert image and use
this box for sizing
Detects abnormal event,
Publishes alert to stdout
Cloudwatch Logs collects
event
Lambda functions triggered to
enhance event metadata, and
notify on abnormal behaviors
Falco at Frame.io
41
https://youtu.be/Z4POV5IXnHQ?t=2648
https://www.usenix.org/node/216834
@mfdii
Join the community.
Website
• https://falco.org
Public Slack
• http://slack.sysdig.com/
• https://sysdig.slack.com/messages/falco
Blog
• https://sysdig.com/blog/tag/falco/
Kubernetes Response Engine
• https://github.com/falcosecurity/kubernetes-
response-engine
Github
• https://github.com/falcosecurity/falco
Documentation
• https://falco.org/docs
Docker Hub
• https://hub.docker.com/r/falcosecurity/falco/
Takeaways
@mfdii
• Easily write simple functions to react to operational events
• Infrastructure available via an API
• Multiple subscribers can take multiple actions
• One function to delete a pod
• One function to notify teams
• One function to log events
• One function to block traffic
• Small, reusable components
• No need to run infrastructure for Operations tasks
Serverless for operations.
@mfdii
• Open Source frameworks are too early, heavy,
incomplete. Use Cloud Providers if possible.
• Automation “runbooks”/”frameworks” don’t exist.
• Need a common library of repeatable tasks.
Challenges in Serverless for Ops
Thank you!
@mfdii
@mfdii

Automating Security Response with Serverless

  • 1.
  • 2.
    @mfdii About me. Michael Ducy CloudNative Security & Transformation, Red Hat Past: Sysdig, Chef, Orbitz WW DevOpsDays Columbus Organizer Cloud Native Security Day Organizer https://events.linuxfoundation.org/cloud-native-security-day/ @mfdii @mfdii
  • 3.
    @mfdii • Serverless Overview •Security Response with Serverless • MLGuard • Falco and Security Playbooks • Take Aways Agenda
  • 4.
    @mfdii • Faster timeto production. • No servers to manage. • Fine grained metering. • Continuous scaling. • Easily integrate with cloud infrastructure. Serverless Benefits
  • 5.
    @mfdii Event Driven: • Triggersa function based on an event • Typically message queue, pub/sub, http webhook Periodic Execution: • Executes a function based on time (Hourly, Daily, etc) Serverless Design Patterns
  • 6.
  • 7.
    @mfdii Knative Kubernetes Native Serverless •Support for any language in a container. • Heavy focus on deployment versioning, blue/green deployments, scaling. • Triggers: Kafka, Google PubSub, Amazon PubSub, CronJob, GitHub, Kubernetes, Camel, Containers. 7
  • 8.
    @mfdii Kubeless: Simple FaaS SimpleFunctions as a Service (Serverless) for Kubernetes: • Support for Python, Node.js, Ruby, PHP, Golang, .NET, Ballerina and custom runtimes • Event triggers using Kafka, NATS, and HTTP events • Prometheus monitoring of functions calls and function latency by default • https://kubeless.io/ 8
  • 9.
    @mfdii • NodeJS, Go,Java, Scala, PHP, Python, Ruby, Swift, Ballerina, .NET and Rust. • Triggers limited: Cloudant, GitHub Apache OpenWhisk
  • 10.
  • 11.
  • 12.
  • 13.
    @mfdii MLGuard • Overcomes limitationsof Web Application Firewalls • WAFs are static and easy to evade: Static & Signature Based • MLGuard provides anomaly detection • Serverless-based machine learning system to detect malicious web requests. • Developed by Frame.io https://www.youtube.com/watch?v=T7C0qtA8Mww
  • 14.
    @mfdii Profile an IPbased on kinds of http request that it sends MLGuard Concept 192.168.xxx.xx { 2xx: 49 3xx: 0 4xx: 1 5xx: 0 } 192.169.xxx.xx { 2xx: 4 3xx: 2 4xx: 35 5xx: 0 } Hmm, that’s good That’s Suspicious
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
    @mfdii A behavioral activitymonitor • Detects suspicious activity defined by a set of rules • Uses sysdig’s flexible and powerful filtering expressions With full support for containers/orchestration • Utilizes sysdig’s container & orchestrator support And flexible notification methods • Alert to files, standard output, syslog, programs Open Source • Anyone can contribute rules or improvements Falco.
  • 22.
    @mfdii • Containers areisolated processes. • Processes are scoped as to what’s expected. • Container images are immutable, runtime environments often aren’t. • How do you detect abnormal behavior? Anomaly detection.
  • 23.
  • 24.
  • 25.
    @falco_org Falco Architecture FILTER EXPRESSION SYSDIG OSS LIBRARIES FALCO_PROBE KERNELMODULE or eBPF PROBE ALERTING FALCO RULES WEB SERVER USER KERNEL EVENTS SUSPICIOUS EVENTS SHELL STDOUT FILE SYSLOG HTTPs gRPC KUBERNETES AUDIT LOGS & METADATA FALCO client-go FALCO SIDEKICK
  • 26.
  • 27.
    @mfdii - list: bin_dirs items:[/bin, /sbin, /usr/bin, /usr/sbin] - macro: bin_dir condition: fd.directory in (bin_dirs) - rule: write_binary_dir desc: an attempt to write to any file below a set of binary directories condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs output: "File below a known binary directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)" priority: WARNING yaml file containing Macros, Lists, and Rules Falco rules.
  • 28.
    @mfdii fd - FileDescriptors process - Processes evt - System events user - Users group - Groups syslog - Syslog messages Based on “Field Classes”. Supported classes include: container - Container metadata fdlist - FD poll events k8s - Kubernetes metadata ka - Kubernetes Audit Logs mesos - Mesos metadata Conditions and Sysdig Filter Expressions.
  • 29.
    @mfdii Quick examples. evt.type =setns and not proc.name in (docker, sysdig) fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin) and write container.id != host and proc.name = bash evt.type = open and fd.name = /dev/video0 and not proc.name in (skype, webex) (evt.type = create or evt.arg.flags contains O_CREAT) and proc.name != blkid and fd.directory = /dev and fd.name != /dev/null A shell is run in a container Overwrite system binaries Container namespace change Non-device files written in /dev Process tries to access camera
  • 30.
    @mfdii Sending alerts • Eventsmatching filter expression result in alerts • Rule’s output field used to format event into alert message • Falco configuration used to control where alert message is sent Any combination of.. • GRPC • Syslog • File • Standard Output • Shell (e.g. mail -s "Falco Notification" someone@example.com) Alerts + outputs.
  • 31.
    @mfdii - rule: NodeContainer Runs Node desc: Detect a process that’s not node started in a Node container. condition: evt.type=execve and container.image startswith node and proc.name!=node output: Node container started other process (user=%user.name command=%proc.cmdline %container.info) priority: INFO tags: [container, apps] A custom Falco rule. Something is executing a program In a container based on the Node image And the process name isn’t node
  • 32.
    @mfdii - macro: contains_private_credentials condition:> (ka.req.configmap.obj contains "aws_access_key_id" or ka.req.configmap.obj contains "aws_s3_access_key_id" or ka.req.configmap.obj contains “password") - macro: configmap condition: ka.target.resource=configmaps - macro: modify condition: (ka.verb in (create,update,patch)) - rule: Create/Modify Configmap With Private Credentials desc: Detect creating/modifying a configmap containing a private credential (aws key, password, etc.) condition: configmap and modify and contains_private_credentials output: K8s configmap with private credential (user=%ka.user.name verb=%ka.verb name=%ka.req.configmap.name configmap=%ka.req.configmap.name config=%ka.req.configmap.obj) priority: WARNING source: k8s_audit tags: [k8s] K8s audit log rule example.
  • 33.
  • 34.
    @mfdii Installing Falco. • DebianPackage • apt-get -y install falco • Redhat Package • yum -y install falco • Installation Script • curl -s s3.amazonaws.com/download.draios.com/stable/install-falco | sudo bash • Docker container • docker pull sysdig/falco • Full instructions • github.com/draios/falco/wiki/How-to-Install-Falco-for-Linux
  • 35.
    @mfdii • Use Helm •$ helm install --name sysdig-falco-1 stable/falco • https://sysdig.com/blog/falco-helm-chart/ • Install Falco as Kubernetes Daemonset • https://github.com/draios/falco/tree/dev/examples/k8s-using-daemonset • Configuration stored in Kubernetes ConfigMaps • Conditions in a Falco Rule can leverage Kubernetes metadata to trigger events • Falco events can include Kubernetes metadata to give notification context: • name, id, labels for Pods, ReplicationController, Service, Namespace, ReplicaSet, and Deployment Installing Falco on kubernetes.
  • 36.
  • 37.
    @mfdii • Detect abnormalevents with Falco • Publish alerts to Pub/Sub service (NATS.io) • Subscribers can subscribe to various FALCO topics to receive alerts: • FALCO.* - All alerts • FALCO.Notice - Alerts of priority “Notice” only • FALCO.Critical - Alerts of priority “Critical” only • Subscribers can take action on alerts: • Kill offending Pod • Taint Nodes to prevent scheduling • Isolate Pod with Networking Policy • Send notification via Slack Response engine & security playbooks.
  • 38.
    @mfdii Image Placeholder Insert imageand use this box for sizing Image Placeholder Insert image and use this box for sizing Image Placeholder Insert image and use this box for sizing Detects abnormal event, Publishes alert to NATS Subscribers receive Falco Alert through NATS Server Kubeless receives Falco Alert, firing a function to delete the offending Kubernetes Pod https://sysdig.com/blog/oss-container-security-runtime/ Response engine & security playbooks.
  • 39.
    @mfdii Response engine & securityplaybooks. https://sysdig.com/blog/container-security-orchestration-falco-splunk-phantom/ PUBLISH TO TOPIC FALCO-NATS SIDECAR LINUX PIPE FALCO CONTAINER FALCO DAEMONSET EVENTS K8S METADATA KUBERNETES NODES kubelet API APPLICATION DEPLOYMENTS EXECUTE REACTION i.e. kill the offending pod F(x) F(x) F(x) WEBHOOK NOTIFICATION SUBSCRIBE TO 1..N TOPICS kubernetes
  • 40.
    @mfdii Response engine & securityplaybooks. PUBLISH TO TOPIC FALCO-NATS SIDECAR LINUX PIPE FALCO CONTAINER FALCO DAEMONSET EVENTS K8S METADATA KUBERNETES NODES kubelet API APPLICATION DEPLOYMENTS EXECUTE REACTION i.e. kill the offending pod F(x) F(x) F(x) WEBHOOK NOTIFICATION SUBSCRIBE TO 1..N TOPICS AWS LAMBDA https://aws.amazon.com/blogs/opensource/securing-amazon-eks-lambda-falco/
  • 41.
    @mfdii Image Placeholder Insert imageand use this box for sizing Image Placeholder Insert image and use this box for sizing Image Placeholder Insert image and use this box for sizing Detects abnormal event, Publishes alert to stdout Cloudwatch Logs collects event Lambda functions triggered to enhance event metadata, and notify on abnormal behaviors Falco at Frame.io 41 https://youtu.be/Z4POV5IXnHQ?t=2648 https://www.usenix.org/node/216834
  • 42.
    @mfdii Join the community. Website •https://falco.org Public Slack • http://slack.sysdig.com/ • https://sysdig.slack.com/messages/falco Blog • https://sysdig.com/blog/tag/falco/ Kubernetes Response Engine • https://github.com/falcosecurity/kubernetes- response-engine Github • https://github.com/falcosecurity/falco Documentation • https://falco.org/docs Docker Hub • https://hub.docker.com/r/falcosecurity/falco/
  • 43.
  • 44.
    @mfdii • Easily writesimple functions to react to operational events • Infrastructure available via an API • Multiple subscribers can take multiple actions • One function to delete a pod • One function to notify teams • One function to log events • One function to block traffic • Small, reusable components • No need to run infrastructure for Operations tasks Serverless for operations.
  • 45.
    @mfdii • Open Sourceframeworks are too early, heavy, incomplete. Use Cloud Providers if possible. • Automation “runbooks”/”frameworks” don’t exist. • Need a common library of repeatable tasks. Challenges in Serverless for Ops
  • 46.