Detecting Data Exfiltration on the Edge with Pixie

1. Detecting Data Exfiltration on the Edge with Pixie
Zain Asgar - GM/GVP - Pixie & Open Source - New Relic
Omid Azizi - Principal Engineer - Pixie & Open Source - New Relic

2. @oazizi
Principal engineer at New Relic.
Founding engineer at Pixie Labs (@pixie_run)
Hi
@zainasgar
GVP & GM at New relic
Co founder/CEO of Pixie Labs (@pixie_run) &
Adjunct Professor of CS @ Stanford

3. Disclaimer
We are not security experts.
The contents of this talk are not meant for use in production.
Our goal is to demonstrate some ideas and start discussions.

4. Data Exfiltration Risks
Data exfiltration is a huge risk.
● Leaks of information like credit cards, SSNs, phone numbers lead to theft.
● Costs $$$ (data?)
Wouldn’t it be great if we could know if sensitive data is leaving your K8s cluster?
● It all starts with observability.

5. Pixie: eBPF-based Observability for K8s
Auto-Telemetry
(eBPF)
Edge
Compute
Scriptable
Interfaces

6. Pixie’s Edge Architecture
● Cloud: Control plane for serving API/UI and
managing metadata
● Vizier: Data plane for collecting and
processing data
○ Deployed to each cluster
○ All data stored in-memory in-cluster
○ All outgoing data is E2E encrypted

7. Pixie is scriptable
● Valid
● Valid
● Built for data analysis and ML
import px
def http_data():
df = px.DataFrame(table='http_events', start_time='-30s')
df.pod = df.ctx['pod']
return df[['pod', 'http_req_path', 'http_resp_latency_ns']]
px.display(http_data())

8. Pixie: Under The Hood

9. PEM Architecture
PEM: Pixie Edge Module.
● The agent that runs on each host.
Stirling: The PEM’s data collector.
● Contains the eBPF collectors.

10. Stirling: Pixie’s Data Collector
Focus for this talk

11. An eBPF probe is like a breakpoint in a debugger.
● Interrupts execution when breakpoint is reached.
Unlike a breakpoint, a small program is run:
● Inspects and collects any relevant state.
● Then immediately resumes the execution.
In BPF terminology, this is called a probe.
● Results in minimized overhead to the execution.
// Small probe
// counts number of
// calls to SendPong()
int ProbeSendPong() {
pong_count++;
}
void ProcessReq(Req r) {
if (r.body == "ping") {
SendPong();
}
...
}
probe
Application Code eBPF Code
What is eBPF? An Analogy.

12. Protocol Tracing
Snoop messages at the kernel syscall.
● Covers all traffic → reduce blind spots.

13. How observability can catch data leaks
Step 1: Use Pixie to trace traffic on K8s.
Step 2: Run a script to find messages that have sensitive information.
● Look for common patterns:
○ Credit Cards: xxxx-xxxx-xxxx-xxxx
○ SSNs: xxx-xx-xxxx
○ More: Emails, MAC addresses, IP addresses, …
● Filter to traffic egressing K8s
Step 3: Scrutinize the egress of sensitive traffic to see if it is legitimate

14. Demo

15. Thank you!...Questions?
Website: px.dev
Github: pixie-io/pixie