Rutgers Cloud Seminar 2017

Operational Visibility and Analytics
Designed for Cloud
Canturk Isci
IBM Research, NY
@canturkisci
Cloud Foundations
Rutgers
Dec 2017
CloudSightResearch
Vulnerability Advisor

Seminar Info
ABSTRACT: We are witnessing a major shift in the foundations of computing. There is an accelerating growth in the scale
and the variety of cloud platforms, emerging runtimes, and new programming models. Cloud computing discussion has
shifted from mere utility and density to cloud-native services and design patterns. Emerging cloud services allow users to
define and provision complex, distributed systems with unprecedented simplicity and agility. With the push of a button entire
stacks of software can be instantiated within minutes with various configurations and customizations. Automation, continuous
integration and delivery further simplify the entire lifecycle management of modern born-on-the-cloud applications. However,
these advances also bring in new research challenges. Operational visibility into the complex, distributed applications,
cloud runtimes and the underlying infrastructure is becoming a persistent pain point across end-users and providers for
operational and security insights. As system and configuration complexity grows, data-driven operational analytics for
security, compliance, configuration and resource management emerge as key areas of new innovation, where traditional
solutions remain ineffective or insufficient.
In this talk I will present an overview of the cloud evolution, emerging runtimes and design patterns. I will describe the
challenges arising from this evolution and where existing techniques fall short. I will then present our recent work on cloud
operational visibility and analytics that tackles some of these challenges. I will describe a unique approach to leveraging
cloud abstractions and implementation principles to achieve unmatched deep and seamless visibility into cloud instances,
and using this deep visibility to develop operational and security analytics for the cloud. I will overview two outcomes of this
approach, Agentless System Crawler and the Vulnerability Advisor service. I will discuss our journey developing the
foundations of the visibility and security services for IBM Cloud Containers. I will share our experiences working with a
production cloud and the key real-world use cases. I will provide an overview of our current research directions, open
problems and opportunities in this area.

Introductions
EEE VLSI
DSP
Comp Arch
Power Eff
DRS DPM DC, Energy, Virt,
Cloud, Sec., Analytics
Virtualization RM Datacenter Energy
and Thermal
Server Power
States (S3)
Data Center
Robots
Systems as Data Cloud OpVis &
Sec Analytics
Vulnerability
Advisor

Introductions++
Cloud Foundations and Programming Models: From Infrastructure to ML/DL and Serverless
Next-gen Cloud Infrastructure
(HW, Netw, DC, Accelarators,…)
Next-gen Cloud Platform
(Containers, Kubernetes, Netw)
Cloud DevOps
(CI/CD, Pipelines, A/B, Canary, Toolchains)
(Deep Introspection, Crawlers and VA)
Serverless Computing
(OpenWhisk)
Microservices Framework
(Istio, Tracing, Analytics)
Deep Learning As a Service
(DLaaS, Watson ML)
API Ecosystems
(API Harmony)
Security
CI
CI
CI

Introductions++
Cloud DevOps
(OpenWhisk)
(DLaaS, Watson ML)
API Ecosystems
(API Harmony)
Security
CI
CI
CI

Next Time
(DLaaS, Watson ML)
API Ecosystems
(API Harmony)
Security
CI
CI
CI
Cloud DevOps
(OpenWhisk)

7
Cloud Evolution & Challenges
What is Great
 Density
 Scale
 Portability
 Repeatability
 Speed
What Needs Work
 Visibility
 Operational Insight
 Cloud-native Security
Utility Cost Scale
Automatio
n
Agility
(u)Service
s
Dev/Sec/Ops
Intelligence
- Modernization of IT infra and SW delivery
- Complex made simple
- Unprecedented efficiency and TTV
- Lots of shiny toys across IT lifecycle
- Visibility into our environments remains an issue
- Also lots of shiny toys for monitoring & analytics
BUT:
- Still, mostly based on traditional IT Principles!

Traditional, Custom, Siloed Visibility Built into Each Platform + Service
Runtime
Runtime
Runtime
Runtime
Runtime
Runtime
10

RuntimeRuntime
Runtime
Runtime
RuntimeRuntime
Runtime
Runtime
Runtime
Runtime
Runtime
Runtime
Runtime
Runtime
We Get Carried Over:
Verticals Bake Their Own Police, Doctor, Auditor to Each Endpoint
11

Then This Happens:
We Look at Everything, We Cannot Deduce Anything

We Need a New Approach: Built-in Visibility+ Systems as Data + Shared Analytics
Where We Are
We cannot keep building
visibility into each endpoint
Silos do not work
This does not scale
and is not manageable
Where We Need to Go
Visibility built into the cloud
instead of each endpoint
Treat every system as data (cloud makes possible)
Build analytics over data
Use introspection at scale
Systems > Data > Scalable Analytics (aaS)

14
- Provide unmatched deep, seamless visibility into cloud instances
- Drive operational insights to solve real-world pain points
Our Work: Built-in Visibility+ Systems as Data + Shared Analytics

15
- Provide unmatched deep, seamless visibility into cloud instances
Built-in Operational Visibility & Analytics Designed for Cloud

16
- Provide unmatched deep, seamless and unified visibility into ALL cloud instances
Built-in Operational Visibility & Analytics Designed for Cloud
Agentless System Crawler (ASC)

17
Traditional Monitoring/Security vs. Crawlers
OS
Host
Wkld
Agent
Agent
Agent
Agent
OS
Host
Wkld A A
AA
VM
OS Wkld A A
AA
Host
OS
Wkld
A A
AA
Cont
. Wkld
A A
AA
Cont
. Wkld
A A
AA
Cont
.
VMBMS Container
OS
Host
Wkld OS
Host
Wkld
VM
OS Wkld
Host
OS
Wkld
Cont
. Wkld
Cont
. Wkld
Cont
.
VMBMS Container

18
”Users do not have to do anything to get this visibility. It is already there by default”
Container Cloud
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Metrics & Logs
Bus
Multitenant
Index
Logmet
Svc
Provisioning
Tenancy Info
State
Events
 Built-in in every compute node, all geos
 Enabled by default for all users in all prod
 O(10K) metrics/s & logs/s
Current State
Seamless: Built-in Operational Visibility for Containers

19
Container Cloud
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Cool!
Happy User:
Effortless, painless
visibility in user world
magic
Seamless: Built-in Operational Visibility for Containers
”Users do not have to do anything to get this visibility. It is already there by default”

20
Key Advantages
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Why Agentless System Crawlers
magic
 Monitoring built into the platform
not in end-user systems
 No complexity to end user
(They do nothing, all they see is the service)
 No agents/credentials/access
(nothing built into userworld)
 Works out of the box
 Makes data consumable*
(lower barrier to data collection and analytics)
 Better Security* for end user
(No attack surface, in userworld)
 Better Availability* of monitoring
(From birth to death, inspect even defunct guest)
 Guest Agnostic
(Build for platform, not each user distro)
 Decoupled* from user context
(No overhead/side-effect concerns)
 Monitoring done right for the
processes of the Cloud OS

21
Crawler: How it Works for VMs
• Leverage VM Introspection (VMI) techniques to access VM Mem and Disk state
(We built bunch or our own optimizations that make this very efficient and practical)
• Can even remote both (decouple all from VM and host)
• Almost no new dependencies on host
• Currently support 1000+ kernel distros
Hypervisor
MEM
View
KB
APP
Analytics
Apps
Memory
Crawl
API
VM
OS
MEMDisk
Disk
View
Disk
Crawl
API
Cloud Analytics
Crawl
Logic Structured
view of
VM states
APP
APP
{
.......
.......
}
Frames

22
Crawler: VM Introspection Modalities
• Crawling VM Images
(IVIL, CMU/UCSD/Lib of Congress Olive)
• Runtime VM Disk Introspection
(RC2,)
• Delta Block Tracking
(Fine-grain, streaming, CMU)
• Runtime VM Memory Introspection
(Live state, near-realtime, UofT)
*
I
II
III

23
Crawler: VM + Container Introspection Modalities
• Container Image Crawling
(IBM Containers, Registry, Toolchain, ICp, Watson)
• Live Container Introspection
(IBM Containers, Registry, Toolchain, ICp, Watson)
• …
IV
V
VI

24
Crawler: How it Works for Containers
1) Get visibility into container world
by namespace mapping
2) Crawl the container
(Crawler dependencies still borrowed from host.
No need to inject into container!)
3) Return to original namespace
4) Push data to backend index
• Leverage Docker APIs for base container information
• Exploit container abstractions (namespace mapping and cgroups) for deeper insight
• Provide deep state info at scale with no visible overheads to end user

25
Crawler: Container Introspection Details (1/2)

26
Crawler: Container Introspection Details (2/2)

Crawler: Performance
• Crawl latency per feature for 200 Cs on host • Crawl overhead for 200 Cs

28
Where We Are Today: Maturity & Contributing & Impact
 Unified Operational Visibility for: [containers, VMs, host] x [x86, PPCle]
 Agentless System Crawler OSS in GitHub, dwOpen
 Deployed in IBM Cloud Containers, Private, Watson, DLaaS
 Externally accessible papers, blogs, podcasts, tech talks

29
Where We Are Going & Open Problems
 Secure Extensibility (Big): Ready for primetime 2018
 Understanding & mitigating side effects: Open
 Bare Metal Image Introspection:
 BU course offering in MOC 2018
 Microservices Operational Visibility:
 Crawlers + Istio Analytics in 2018

 VM Introspection:
 Remoting Mem, Realtime Introspection: Open
 Operational visibility for serverless: ~Open
 Crawling/Mining for PII (+ certs, bitcoin, …): Open
 Privacy-preserving Opvis: Open

3030
Nostalgia: Big Vision = Systems as Data
Transform systems into
documents/frames/data
Crawl the cloud like you crawl
the web
Query & mine the cloud like
query/mine the web
Learn good & bad sytem/SW
configurations automagically

Nostalgia: Big Vision = Analytics over Systems Data

32
Deep Visibility: What We Actually Collect (and Annotate)
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator

33
Deep Visibility  Operational Insights/Analytics  Solve Real Problems
Index (Data)
Vuln. &
Compl.
Analysis
Config
Analytics
(SecConfig)
Cloud Time
Machine
(Audit/PD)
Pipeline
Service
(DevOps)
Remediation
Service
* All analytics services
work from the
same data & pipeline!
Today’s Special:
Vulnerability Advisor- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
Data Collection Curation Index (Data) Analyitcs

34
Operational Analytics Data Pipeline [Where We Started]
Images
(Registry)
Kafka
Configuration Channel
Compliance Channel
Vulnerability Channel
Indexers
Vulnerability Annotator
Elastic
Configuration Index
Compliance Index
Vulnerability Index
Compliance Annotator

35
Operational Analytics Data Pipeline [Where We Are]
Images
(Registry) Notification Channel
Kafka
Configuration Channel
Compliance Channel
Vulnerability Channel
Indexers
Vulnerability Annotator
Discovery Channel
Instances
(Compute) SecConfig Channel
Rootkit Channel
Licence Channel
Notification Index
Elastic
Configuration Index
Compliance Index
Vulnerability Index
Discovery Index
SecConfig Index
Rootkit Index
Licence Index
USNs Index
Compliance Annotator
Password Annotator
Config Parser
SecConfig Annotator
SW Discovery
Rootkit Annotator
Licence Discovery
Notification Parser
External
Feeds

Operational Analytics Growth
In Cloud Svc
In Research
Docker Hosts
App
Cont.
App
Cont.
App
Cont.
App
Cont.
Docker Hosts
App
Cont.
App
Cont.
App
Cont.
App
Cont.
Compute Hosts
App
Cont.
App
Cont.
App
Cont.
App
Cont.
Image Registry
IMG IMG IMG IMG
IMG IMG IMG IMG
IMG
x86 PPCLE
Containers
Images

Vulnerability Advisor Today
• f(Platform Visibility)  Security Analytics
• Slice of data-driven analytics leveraging platform visibility
• Service in IBM Containers, Built-in feature in IBM Cloud Private, Used by Watson, DevOps Toolchains,…
• Security and compliance insight into images and running containers
• Basic policy management and enforcement

Research Beyond Vulnerability Advisor
Config Analytics [ICDCS’18?]
App Security Configs [MW’17]Binary Analysis w Signatures [BigData]
ML for SecBot & Doc Conversion
Cloud Drift Analysis [ATC’17]
Custom Rules and Policy Management
CIS Benchmarks
IBM Apps
SW & System Discovery [BigData,IC2E]
Threat/Access/Behavior Analytics
Black-box Analytics [ICDCS’18?]

Software Discovery and Product Sec. Analysis (Columbus + PSIRT)
• Discover what is in the box using content analysis & discovery
• Columbus: Content labeling based on filesystem impression
92%
Successful
Failed -
BaseImag
es

41
Summary & Parting Thoughs
Summary:
 Challenges: Operational visibility into all systems; need shared operational intelligence
 Opportunities: Systems as data; Data-driven analytics; Many low-hanging problems
 Agentless System Crawler and Vulnerability Advisor are first examples
 New innovations: Cloud + Analytics + Systems + Security
Parting Thoughts:
 Operational Visibility >> Metrics & Logs
 Treat systems as just another form of data
 Cloud enables novel, scalable “monalytics”
 New cloud OS, runtimes, programming models and orchestration frameworks
create many new challenges: Provenance; Security, Accounting,
 No one is immune: Cloud OS spans all verticals:
AI, IoT, Health, Finance, Security, Transport, …
 Security and compliance are coming strong
Privacy and data sensitivity need analytics-driven solutions
 We need to break silos of intelligence

Learn More: Open Innovation <3
Operational Visibility / Agentless System Crawler
Web/Blogs: https://www.google.com.tr/search?q=%22agentless+system+crawler%22
DwOpen: https://developer.ibm.com/open/agentless-system-crawler/
Code/Contribute:https://github.com/cloudviz/agentless-system-crawler
Run/DockerHub: https://hub.docker.com/u/cloudviz/
TechTalk: https://developer.ibm.com/open/videos/agentless-system-crawler-tech-talk/
SlideShare: http://www.slideshare.net/canturkisci/
YouTube: https://www.youtube.com/channel/UCf8Fn8dKQzBCJRgI1jOlGYg
Podcast: https://soundcloud.com/thenewstackmakers/creating-analytics-driven-solutions-for-operational-visibility
Pubs: http://canturkisci.com/ETC/MYpublications.html
(MW’17, ICDCS’17, IC2E’17, VEE’17, HotCloud’17, JRD’16, HotCloud’15, VEE’15, Sigm’14, IC2E’14)
Operational Analytics / Vulnerability Advisor
Web/Blogs: https://www.google.com.tr/search?q=%22vulnerability+advisor%22+ibm
Run: IBM Containers & IBM Cloud Private
SlideShare: http://www.slideshare.net/canturkisci/
YouTube: https://www.youtube.com/channel/UCf8Fn8dKQzBCJRgI1jOlGYg
Pubs: http://canturkisci.com/ETC/MYpublications.html
(MW’17, ATC’17, JRD’16, BigData’16, BigData’14, Sigmetrics’14)
Vulnerability Advisor

Thank You
Operational Visibility and Analytics Designed fro Cloud
[feat. Agentless System Crawler & Vulnerability Advisor]
IBM Research
Cloud Monitoring, Operational and Security Analytics
http://www.canturkisci.com/blog
@canturkisci

44
Crawler: Typical Deployment
• Typical deployment, able to track diverse cloud runtimes w parity
• Need not be on same host, most crawler functions can be even remoted

45
Crawler: Design
• Same crawler across runtimes for unified operational visibility
• Multiple fanouts as use cases grow

46
DEMO TIME
This Session
 Agentless System Crawler
 Bluemix Test Drive (live – ldwave)
https://developer.ibm.com/bluemix/2015/11/16/
built-in-monitoring-and-logging-for-bluemix-containers/
 LogCrawler and JSON Parsing
(live – CanoLibUK3)
 Vanilla LogCrawler
(20150619_LogCrawlerDemo)
 Crawl even Non-responsive systems
(oopsRconsole2)
 Out of Band SIEM
(QRadarDemo)
 TopoLog for Topology Discovery
(newTopo)
 RTop for Realtime Monitoring
(RtopAnnotatedMOV)
 Crawling for Rootkits with RConsole
(RConsoleAnnotatedMOV)
Sunday & Wednesday
 Vulnerability Advisor
 Coming soon…

47
Bluemix Test Drive
Just start a Bluemix Container
(https://console.ng.bluemix.net/)
Go to Container Overview
(Metrics show up in few mins)

48
… Bluemix Test Drive
Go to Monitoring and Logs
>> Monitoring

49
… Bluemix Test Drive
Go to Monitoring and Logs
>> Logging

50
DEMO TIME
This Session
 Vulnerability Advisor, Policy Mgr
 Go to Bluemix Catalog
 See VA Image Status
(Safe, Caution, Blocked)
 Go to Create View
 Explore Status Details
(Vulnerabilities, Policy Violations)
 Browse Policy Manager
(Policy Settings, Deployment Impact)
 Change Org Policies
 Override Policies
(Don’t do it)
 See Weak Password Discovery
 Update Image in Local Dev
 Fix Policy Violation
Previously
 Built-in Monitoring & Logging
 We just did that one…

51
Getting Started: Let’s Go to London
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)

52
Deployment Status
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy

53
Deployment Status
Safe to Deploy | Deploy with Caution

54
Deployment Status
Safe to Deploy | Deploy with Caution | Blocked

55
Create View
Click on Image to go to Create View
See Verdict Details and Explore Options

56
Vulnerability Advisor Report
View Vulnerability Advisor Report:
Discovered Vulnerabilities | Policy Violations

57
Vulnerability Advisor Report

58
Policy Manager and Deployment Impact

59
Change Org Policy and Observe Impact

60
Policy Override
Change Org Policy and Observe Impact
Create View > Click One-time Override
Name your risky container and deploy

61
Also: One-stop Shop “Michael View” for the Purists

Our Journey: Deep Visibility  Operational Analytics  Solve Real Problems
Crawlers Debut:
Built-in workload
visibility
[InterConnect]
VA Debut:
Vuln. and Compl.
scanning for images
[DockerCon]
Weak PW
Detection
Policy
Management
[DockerConEU]
Feb 2015 Jul 2015 Nov 2015 2Q 2016 3Q 2016 4Q 2016
VA for Live
Containers
[DockerCon]
VA for Power
Images
[DockerCon]
Secure Config
Advisor for
Applications
Malware
Discovery
Risk Analysis with
IBM XForce

66
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
How can I identify my vulnerable/non-compliant images
before they go live?
How can I detect and block systems with password access
configurations and weak passwords?
66

67
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
How can I track, query and analyze my configurations in a simple
and robust manner for drift/config analytics?
How can I do better resource management and allocation?
67

Open Challenges: SW Provenance & Registry Sprawl
Ubuntu Repo
RHEL Repo
Mysql.com
Vulnerability Assessment
License Management
Advanced search
Traditional Software
delivery
New age software
delivery PaaS Cloud Platforms

Open Challenges: SW Delivery Modalities
Software packaging and installation
OS
Linux MacOS Windows
RHEL
apt/dpkg
Ubuntu/
debain
yum/rpm
homebrew Installer,
oneGET
Language
python
pip easy_install
ruby
gem
go
go get
javascript
npm
virtualization
Virtual
Machine
Application
images
eg.AWS AMI
containers
Container
app images
eg. Docker image
unikernels
opam
Install from
source
make/
make install
precompiled
binary copy
• Software delivery techniques have exploded and still
growing
• Require separate techniques for each installation path,
making software discovery complex & harder

Open Challenges: Multiple Runtimes/Paracloud
Application Development Application Deployments
Desktops/Servers
IaaS Cloud
PaaS Cloud
Applications mostly have been oblivious to their actual
runtimes

Open Challenges: Prod Visibility/Debuggability on the Fly (CIVIC)
• Analogy: Inspecting a car while running
• 1. Safe reuse of monitoring agents
• 2. High overhead profiling, prohibitive to do on production system
• Attaching intrusive anomaly detector to live service
• 3. Impact-heavy problem diagnostics
• Memory leak troubleshooting
• 4. Workload-specific configuration tuning
• Live webserver reconfiguration

Open Challenge: Understanding and Improving Resource
Accounting for Containers and Other Emerging Runtimes
• Unlike VMs and Servers, new runtimes offer poor isolation of resources
• Many challenges in proper accounting and resource mgmt

Rutgers Cloud Seminar 2017

More Related Content

What's hot

Similar to Rutgers Cloud Seminar 2017

Recently uploaded

Rutgers Cloud Seminar 2017