1. Operational Visibility and Analytics
Designed for Cloud
Canturk Isci
IBM Research, NY
@canturkisci
Cloud Foundations
Rutgers
Dec 2017
CloudSightResearch
Vulnerability Advisor
2. Operational Visibility and Analytics
Designed for Cloud
Canturk Isci
IBM Research, NY
@canturkisci
Cloud Foundations
Rutgers
Dec 2017
CloudSightResearch
Vulnerability Advisor
3. Seminar Info
ABSTRACT: We are witnessing a major shift in the foundations of computing. There is an accelerating growth in the scale
and the variety of cloud platforms, emerging runtimes, and new programming models. Cloud computing discussion has
shifted from mere utility and density to cloud-native services and design patterns. Emerging cloud services allow users to
define and provision complex, distributed systems with unprecedented simplicity and agility. With the push of a button entire
stacks of software can be instantiated within minutes with various configurations and customizations. Automation, continuous
integration and delivery further simplify the entire lifecycle management of modern born-on-the-cloud applications. However,
these advances also bring in new research challenges. Operational visibility into the complex, distributed applications,
cloud runtimes and the underlying infrastructure is becoming a persistent pain point across end-users and providers for
operational and security insights. As system and configuration complexity grows, data-driven operational analytics for
security, compliance, configuration and resource management emerge as key areas of new innovation, where traditional
solutions remain ineffective or insufficient.
In this talk I will present an overview of the cloud evolution, emerging runtimes and design patterns. I will describe the
challenges arising from this evolution and where existing techniques fall short. I will then present our recent work on cloud
operational visibility and analytics that tackles some of these challenges. I will describe a unique approach to leveraging
cloud abstractions and implementation principles to achieve unmatched deep and seamless visibility into cloud instances,
and using this deep visibility to develop operational and security analytics for the cloud. I will overview two outcomes of this
approach, Agentless System Crawler and the Vulnerability Advisor service. I will discuss our journey developing the
foundations of the visibility and security services for IBM Cloud Containers. I will share our experiences working with a
production cloud and the key real-world use cases. I will provide an overview of our current research directions, open
problems and opportunities in this area.
4. Introductions
EEE VLSI
DSP
Comp Arch
Power Eff
DRS DPM DC, Energy, Virt,
Cloud, Sec., Analytics
Virtualization RM Datacenter Energy
and Thermal
Server Power
States (S3)
Data Center
Robots
Systems as Data Cloud OpVis &
Sec Analytics
Vulnerability
Advisor
5. Introductions++
Cloud Foundations and Programming Models: From Infrastructure to ML/DL and Serverless
Next-gen Cloud Infrastructure
(HW, Netw, DC, Accelarators,…)
Next-gen Cloud Platform
(Containers, Kubernetes, Netw)
Cloud DevOps
(CI/CD, Pipelines, A/B, Canary, Toolchains)
Operational Visibility and Analytics
(Deep Introspection, Crawlers and VA)
Serverless Computing
(OpenWhisk)
Microservices Framework
(Istio, Tracing, Analytics)
Deep Learning As a Service
(DLaaS, Watson ML)
API Ecosystems
(API Harmony)
Security
CI
CI
CI
6. Introductions++
Cloud Foundations and Programming Models: From Infrastructure to ML/DL and Serverless
Next-gen Cloud Infrastructure
(HW, Netw, DC, Accelarators,…)
Cloud DevOps
(CI/CD, Pipelines, A/B, Canary, Toolchains)
Serverless Computing
(OpenWhisk)
Microservices Framework
(Istio, Tracing, Analytics)
Deep Learning As a Service
(DLaaS, Watson ML)
API Ecosystems
(API Harmony)
Security
CI
Operational Visibility and Analytics
(Deep Introspection, Crawlers and VA)
CI
Next-gen Cloud Platform
(Containers, Kubernetes, Netw)
CI
7. Next Time
Cloud Foundations and Programming Models: From Infrastructure to ML/DL and Serverless
Next-gen Cloud Infrastructure
(HW, Netw, DC, Accelarators,…)
Deep Learning As a Service
(DLaaS, Watson ML)
API Ecosystems
(API Harmony)
Security
CI
Operational Visibility and Analytics
(Deep Introspection, Crawlers and VA)
CI
Next-gen Cloud Platform
(Containers, Kubernetes, Netw)
CI
Cloud DevOps
(CI/CD, Pipelines, A/B, Canary, Toolchains)
Microservices Framework
(Istio, Tracing, Analytics)
Serverless Computing
(OpenWhisk)
8. 7
Cloud Evolution & Challenges
What is Great
Density
Scale
Portability
Repeatability
Speed
What Needs Work
Visibility
Operational Insight
Cloud-native Security
Utility Cost Scale
Automatio
n
Agility
(u)Service
s
Dev/Sec/Ops
Intelligence
- Modernization of IT infra and SW delivery
- Complex made simple
- Unprecedented efficiency and TTV
- Lots of shiny toys across IT lifecycle
- Visibility into our environments remains an issue
- Also lots of shiny toys for monitoring & analytics
BUT:
- Still, mostly based on traditional IT Principles!
14. We Need a New Approach: Built-in Visibility+ Systems as Data + Shared Analytics
Where We Are
We cannot keep building
visibility into each endpoint
Silos do not work
This does not scale
and is not manageable
Where We Need to Go
Visibility built into the cloud
instead of each endpoint
Treat every system as data (cloud makes possible)
Build analytics over data
Use introspection at scale
Systems > Data > Scalable Analytics (aaS)
15. 14
- Provide unmatched deep, seamless visibility into cloud instances
- Drive operational insights to solve real-world pain points
Our Work: Built-in Visibility+ Systems as Data + Shared Analytics
16. 15
- Provide unmatched deep, seamless visibility into cloud instances
- Drive operational insights to solve real-world pain points
Built-in Operational Visibility & Analytics Designed for Cloud
17. 16
- Provide unmatched deep, seamless and unified visibility into ALL cloud instances
- Drive operational insights to solve real-world pain points
Built-in Operational Visibility & Analytics Designed for Cloud
Agentless System Crawler (ASC)
18. 17
Traditional Monitoring/Security vs. Crawlers
OS
Host
Wkld
Agent
Agent
Agent
Agent
OS
Host
Wkld A A
AA
VM
OS Wkld A A
AA
Host
OS
Wkld
A A
AA
Cont
. Wkld
A A
AA
Cont
. Wkld
A A
AA
Cont
.
VMBMS Container
OS
Host
Wkld OS
Host
Wkld
VM
OS Wkld
Host
OS
Wkld
Cont
. Wkld
Cont
. Wkld
Cont
.
VMBMS Container
19. 18
”Users do not have to do anything to get this visibility. It is already there by default”
Container Cloud
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Docker Hosts
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Metrics & Logs
Bus
Multitenant
Index
Logmet
Svc
Provisioning
Tenancy Info
State
Events
Built-in in every compute node, all geos
Enabled by default for all users in all prod
O(10K) metrics/s & logs/s
Current State
Seamless: Built-in Operational Visibility for Containers
21. 20
Key Advantages
App
Cont
.App
Cont
.App
Cont
.App
Cont
.
Why Agentless System Crawlers
magic
Monitoring built into the platform
not in end-user systems
No complexity to end user
(They do nothing, all they see is the service)
No agents/credentials/access
(nothing built into userworld)
Works out of the box
Makes data consumable*
(lower barrier to data collection and analytics)
Better Security* for end user
(No attack surface, in userworld)
Better Availability* of monitoring
(From birth to death, inspect even defunct guest)
Guest Agnostic
(Build for platform, not each user distro)
Decoupled* from user context
(No overhead/side-effect concerns)
Monitoring done right for the
processes of the Cloud OS
22. 21
Crawler: How it Works for VMs
• Leverage VM Introspection (VMI) techniques to access VM Mem and Disk state
(We built bunch or our own optimizations that make this very efficient and practical)
• Can even remote both (decouple all from VM and host)
• Almost no new dependencies on host
• Currently support 1000+ kernel distros
Hypervisor
MEM
View
KB
APP
Analytics
Apps
Memory
Crawl
API
VM
OS
MEMDisk
Disk
View
Disk
Crawl
API
Cloud Analytics
Crawl
Logic Structured
view of
VM states
APP
APP
{
.......
.......
}
Frames
23. 22
Crawler: VM Introspection Modalities
• Crawling VM Images
(IVIL, CMU/UCSD/Lib of Congress Olive)
• Runtime VM Disk Introspection
(RC2,)
• Delta Block Tracking
(Fine-grain, streaming, CMU)
• Runtime VM Memory Introspection
(Live state, near-realtime, UofT)
*
I
II
III
24. 23
Crawler: VM + Container Introspection Modalities
• Container Image Crawling
(IBM Containers, Registry, Toolchain, ICp, Watson)
• Live Container Introspection
(IBM Containers, Registry, Toolchain, ICp, Watson)
• …
IV
V
VI
25. 24
Crawler: How it Works for Containers
1) Get visibility into container world
by namespace mapping
2) Crawl the container
(Crawler dependencies still borrowed from host.
No need to inject into container!)
3) Return to original namespace
4) Push data to backend index
• Leverage Docker APIs for base container information
• Exploit container abstractions (namespace mapping and cgroups) for deeper insight
• Provide deep state info at scale with no visible overheads to end user
29. 28
Where We Are Today: Maturity & Contributing & Impact
Unified Operational Visibility for: [containers, VMs, host] x [x86, PPCle]
Agentless System Crawler OSS in GitHub, dwOpen
Deployed in IBM Cloud Containers, Private, Watson, DLaaS
Externally accessible papers, blogs, podcasts, tech talks
30. 29
Where We Are Going & Open Problems
Secure Extensibility (Big): Ready for primetime 2018
Understanding & mitigating side effects: Open
Bare Metal Image Introspection:
BU course offering in MOC 2018
Microservices Operational Visibility:
Crawlers + Istio Analytics in 2018
VM Introspection:
Remoting Mem, Realtime Introspection: Open
Operational visibility for serverless: ~Open
Crawling/Mining for PII (+ certs, bitcoin, …): Open
Privacy-preserving Opvis: Open
31. 3030
Nostalgia: Big Vision = Systems as Data
Transform systems into
documents/frames/data
Crawl the cloud like you crawl
the web
Query & mine the cloud like
query/mine the web
Learn good & bad sytem/SW
configurations automagically
33. 32
Deep Visibility: What We Actually Collect (and Annotate)
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
34. 33
Deep Visibility Operational Insights/Analytics Solve Real Problems
Index (Data)
Vuln. &
Compl.
Analysis
Config
Analytics
(SecConfig)
Cloud Time
Machine
(Audit/PD)
Pipeline
Service
(DevOps)
Remediation
Service
* All analytics services
work from the
same data & pipeline!
Today’s Special:
Vulnerability Advisor- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
Data Collection Curation Index (Data) Analyitcs
35. 34
Operational Analytics Data Pipeline [Where We Started]
Images
(Registry)
Kafka
Configuration Channel
Compliance Channel
Vulnerability Channel
Indexers
Vulnerability Annotator
Elastic
Configuration Index
Compliance Index
Vulnerability Index
Compliance Annotator
36. 35
Operational Analytics Data Pipeline [Where We Are]
Images
(Registry) Notification Channel
Kafka
Configuration Channel
Compliance Channel
Vulnerability Channel
Indexers
Vulnerability Annotator
Discovery Channel
Instances
(Compute) SecConfig Channel
Rootkit Channel
Licence Channel
Notification Index
Elastic
Configuration Index
Compliance Index
Vulnerability Index
Discovery Index
SecConfig Index
Rootkit Index
Licence Index
USNs Index
Compliance Annotator
Password Annotator
Config Parser
SecConfig Annotator
SW Discovery
Rootkit Annotator
Licence Discovery
Notification Parser
External
Feeds
38. Vulnerability Advisor Today
• f(Platform Visibility) Security Analytics
• Slice of data-driven analytics leveraging platform visibility
• Service in IBM Containers, Built-in feature in IBM Cloud Private, Used by Watson, DevOps Toolchains,…
• Security and compliance insight into images and running containers
• Basic policy management and enforcement
39. Research Beyond Vulnerability Advisor
Config Analytics [ICDCS’18?]
App Security Configs [MW’17]Binary Analysis w Signatures [BigData]
ML for SecBot & Doc Conversion
Cloud Drift Analysis [ATC’17]
Custom Rules and Policy Management
CIS Benchmarks
IBM Apps
SW & System Discovery [BigData,IC2E]
Threat/Access/Behavior Analytics
Black-box Analytics [ICDCS’18?]
41. Software Discovery and Product Sec. Analysis (Columbus + PSIRT)
• Discover what is in the box using content analysis & discovery
• Columbus: Content labeling based on filesystem impression
92%
Successful
Failed -
BaseImag
es
42. 41
Summary & Parting Thoughs
Summary:
Challenges: Operational visibility into all systems; need shared operational intelligence
Opportunities: Systems as data; Data-driven analytics; Many low-hanging problems
Agentless System Crawler and Vulnerability Advisor are first examples
New innovations: Cloud + Analytics + Systems + Security
Parting Thoughts:
Operational Visibility >> Metrics & Logs
Treat systems as just another form of data
Cloud enables novel, scalable “monalytics”
New cloud OS, runtimes, programming models and orchestration frameworks
create many new challenges: Provenance; Security, Accounting,
No one is immune: Cloud OS spans all verticals:
AI, IoT, Health, Finance, Security, Transport, …
Security and compliance are coming strong
Privacy and data sensitivity need analytics-driven solutions
We need to break silos of intelligence
44. Thank You
Operational Visibility and Analytics Designed fro Cloud
[feat. Agentless System Crawler & Vulnerability Advisor]
IBM Research
Cloud Monitoring, Operational and Security Analytics
http://www.canturkisci.com/blog
@canturkisci
45. 44
Crawler: Typical Deployment
• Typical deployment, able to track diverse cloud runtimes w parity
• Need not be on same host, most crawler functions can be even remoted
46. 45
Crawler: Design
• Same crawler across runtimes for unified operational visibility
• Multiple fanouts as use cases grow
47. 46
DEMO TIME
This Session
Agentless System Crawler
Bluemix Test Drive (live – ldwave)
https://developer.ibm.com/bluemix/2015/11/16/
built-in-monitoring-and-logging-for-bluemix-containers/
LogCrawler and JSON Parsing
(live – CanoLibUK3)
Vanilla LogCrawler
(20150619_LogCrawlerDemo)
Crawl even Non-responsive systems
(oopsRconsole2)
Out of Band SIEM
(QRadarDemo)
TopoLog for Topology Discovery
(newTopo)
RTop for Realtime Monitoring
(RtopAnnotatedMOV)
Crawling for Rootkits with RConsole
(RConsoleAnnotatedMOV)
Sunday & Wednesday
Vulnerability Advisor
Coming soon…
48. 47
Bluemix Test Drive
Just start a Bluemix Container
(https://console.ng.bluemix.net/)
Go to Container Overview
(Metrics show up in few mins)
51. 50
DEMO TIME
This Session
Vulnerability Advisor, Policy Mgr
Go to Bluemix Catalog
See VA Image Status
(Safe, Caution, Blocked)
Go to Create View
Explore Status Details
(Vulnerabilities, Policy Violations)
Browse Policy Manager
(Policy Settings, Deployment Impact)
Change Org Policies
Override Policies
(Don’t do it)
See Weak Password Discovery
Update Image in Local Dev
Fix Policy Violation
Previously
Built-in Monitoring & Logging
We just did that one…
53. 52
Deployment Status
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy
54. 53
Deployment Status
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution
55. 54
Deployment Status
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution | Blocked
56. 55
Create View
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution | Blocked
Click on Image to go to Create View
See Verdict Details and Explore Options
57. 56
Vulnerability Advisor Report
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution | Blocked
Click on Image to go to Create View
See Verdict Details and Explore Options
View Vulnerability Advisor Report:
Discovered Vulnerabilities | Policy Violations
58. 57
Vulnerability Advisor Report
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution | Blocked
Click on Image to go to Create View
See Verdict Details and Explore Options
View Vulnerability Advisor Report:
Discovered Vulnerabilities | Policy Violations
59. 58
Policy Manager and Deployment Impact
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution | Blocked
Click on Image to go to Create View
See Verdict Details and Explore Options
View Vulnerability Advisor Report:
Discovered Vulnerabilities | Policy Violations
Policy Manager and Deployment Impact
60. 59
Policy Manager and Deployment Impact
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution | Blocked
Click on Image to go to Create View
See Verdict Details and Explore Options
View Vulnerability Advisor Report:
Discovered Vulnerabilities | Policy Violations
Policy Manager and Deployment Impact
Change Org Policy and Observe Impact
61. 60
Policy Override
Login to Bluemix London
(https://console.eu-gb.bluemix.net/)
Go to Catalog and Look for Containers
Hover over containers to see VA verdict:
Safe to Deploy | Deploy with Caution | Blocked
Click on Image to go to Create View
See Verdict Details and Explore Options
View Vulnerability Advisor Report:
Discovered Vulnerabilities | Policy Violations
Policy Manager and Deployment Impact
Change Org Policy and Observe Impact
Create View > Click One-time Override
Name your risky container and deploy
63. Our Journey: Deep Visibility Operational Analytics Solve Real Problems
Crawlers Debut:
Built-in workload
visibility
[InterConnect]
VA Debut:
Vuln. and Compl.
scanning for images
[DockerCon]
Weak PW
Detection
Policy
Management
[DockerConEU]
Feb 2015 Jul 2015 Nov 2015 2Q 2016 3Q 2016 4Q 2016
VA for Live
Containers
[DockerCon]
VA for Power
Images
[DockerCon]
Secure Config
Advisor for
Applications
Malware
Discovery
Risk Analysis with
IBM XForce
64. Our Journey: Deep Visibility Operational Analytics Solve Real Problems
Crawlers Debut:
Built-in workload
visibility
[InterConnect]
VA Debut:
Vuln. and Compl.
scanning for images
[DockerCon]
Weak PW
Detection
Policy
Management
[DockerConEU]
Feb 2015 Jul 2015 Nov 2015 2Q 2016 3Q 2016 4Q 2016
VA for Live
Containers
[DockerCon]
VA for Power
Images
[DockerCon]
Secure Config
Advisor for
Applications
Malware
Discovery
Risk Analysis with
IBM XForce
65. Our Journey: Deep Visibility Operational Analytics Solve Real Problems
Crawlers Debut:
Built-in workload
visibility
[InterConnect]
VA Debut:
Vuln. and Compl.
scanning for images
[DockerCon]
Weak PW
Detection
Policy
Management
[DockerConEU]
Feb 2015 Jul 2015 Nov 2015 2Q 2016 3Q 2016 4Q 2016
VA for Live
Containers
[DockerCon]
VA for Power
Images
[DockerCon]
Secure Config
Advisor for
Applications
Malware
Discovery
Risk Analysis with
IBM XForce
66. Our Journey: Deep Visibility Operational Analytics Solve Real Problems
Crawlers Debut:
Built-in workload
visibility
[InterConnect]
VA Debut:
Vuln. and Compl.
scanning for images
[DockerCon]
Weak PW
Detection
Policy
Management
[DockerConEU]
Feb 2015 Jul 2015 Nov 2015 2Q 2016 3Q 2016 4Q 2016
VA for Live
Containers
[DockerCon]
VA for Power
Images
[DockerCon]
Secure Config
Advisor for
Applications
Malware
Discovery
Risk Analysis with
IBM XForce
67. 66
Deep Visibility Operational Insights/Analytics Solve Real Problems
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
How can I identify my vulnerable/non-compliant images
before they go live?
How can I detect and block systems with password access
configurations and weak passwords?
66
68. 67
- OS Info
- Processes
- Disk Info
- Metrics
- Network Info
- Packages
- Files
- Config Info
From Container/VM
- Docker metadata
(docker inspect)
- CPU metrics
(/cgroup/cpuacct/)
- Memory metrics
(/cgroup/memory)
- Docker history
Docker Runtime
Config
Annotator
Vulnerability
Annotator
Compliance
Annotator
Password
Annotator
SW
Annotator
Licence
Annotator
How can I track, query and analyze my configurations in a simple
and robust manner for drift/config analytics?
How can I do better resource management and allocation?
67
Deep Visibility Operational Insights/Analytics Solve Real Problems
69. Open Challenges: SW Provenance & Registry Sprawl
Ubuntu Repo
RHEL Repo
Mysql.com
Vulnerability Assessment
License Management
Advanced search
Traditional Software
delivery
New age software
delivery PaaS Cloud Platforms
70. Open Challenges: SW Delivery Modalities
Software packaging and installation
OS
Linux MacOS Windows
RHEL
apt/dpkg
Ubuntu/
debain
yum/rpm
homebrew Installer,
oneGET
Language
python
pip easy_install
ruby
gem
go
go get
javascript
npm
virtualization
Virtual
Machine
Application
images
eg.AWS AMI
containers
Container
app images
eg. Docker image
unikernels
opam
Install from
source
make/
make install
precompiled
binary copy
• Software delivery techniques have exploded and still
growing
• Require separate techniques for each installation path,
making software discovery complex & harder
71. Open Challenges: Multiple Runtimes/Paracloud
Application Development Application Deployments
Desktops/Servers
IaaS Cloud
PaaS Cloud
Applications mostly have been oblivious to their actual
runtimes
72. Open Challenges: Prod Visibility/Debuggability on the Fly (CIVIC)
• Analogy: Inspecting a car while running
• 1. Safe reuse of monitoring agents
• 2. High overhead profiling, prohibitive to do on production system
• Attaching intrusive anomaly detector to live service
• 3. Impact-heavy problem diagnostics
• Memory leak troubleshooting
• 4. Workload-specific configuration tuning
• Live webserver reconfiguration
73. Open Challenge: Understanding and Improving Resource
Accounting for Containers and Other Emerging Runtimes
• Unlike VMs and Servers, new runtimes offer poor isolation of resources
• Many challenges in proper accounting and resource mgmt