• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Cartographer, or Building A Next Generation Management Framework
 

Cartographer, or Building A Next Generation Management Framework

on

  • 1,305 views

Dr. Bobby Krupczak's slides about the Cartographer management agent and the underlying XMP management framework. Presented at the February 10, 2009 meeting of the Atlanta Network and Systems ...

Dr. Bobby Krupczak's slides about the Cartographer management agent and the underlying XMP management framework. Presented at the February 10, 2009 meeting of the Atlanta Network and Systems Management Technical User Group (ANSMTUG).

Statistics

Views

Total Views
1,305
Views on SlideShare
1,300
Embed Views
5

Actions

Likes
1
Downloads
22
Comments
0

2 Embeds 5

http://www.slideshare.net 3
http://jeffgehlbach.com 2

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Cartographer, or Building A Next Generation Management Framework Cartographer, or Building A Next Generation Management Framework Presentation Transcript

    • Cartographer or Building a Next Generation Management Framework Bobby Krupczak Chief Scientist Krupczak.org, LLC [email_address] http://www.krupczak.org/cartographer
    • Overview
      • Background
      • Overview of network mgmt today
      • Cartographer
      • Yet another management framework
      • Software technology
      • Demo
    • Who Am I?
      • BS CISE from UF 1989
      • Worked in industry on SNMP
      • MS CS GaTech 1993
      • Co-founder of Empire Technologies
      • PhD CS GaTech 1997
      • Sold Empire to Concord 1999
      • Krupczak.org 2003
    • Management Model
      • Mgmt info is virtual representation
      • Managers, agents exchange mgmt info
      • Mgmt is therefore:
        • Inspection of
        • Alteration of
        • Creation of
        • Deletion of mgmt info
    • First-Generation
      • Dumb, lightweight (hopefully) agents
      • Heavyweight, complex, smart managers
      • Traditional command-control
      • Scaling becomes issue
      • Analogous to CEO managing entire enterprise
    • 2nd-Generation
      • Push intelligence outwards towards agent
      • Empire/SystemEDGE, RMON
      • Increase scaling, reduce reaction time
      • Some delegation, middle-managers, remote pollers
      • Exception-management, event
      • de-duplication, root-cause
    • 2nd-Generation (continued)
      • Agents still work in isolation (stovepipes)
      • Distribution overhead and agent administrative footprint still non-trivial
      • SNMPv1, v2c, v3 now deployed
      • Agent backlash?
      • CEO now has bank VPs but still manage/controls the enterprise
    • Cartographer
      • Discover, track relationships between components in distributed system
        • Dependencies between network, system, applications
        • Include network services as well as higher-layer abstractions
        • Agent based
        • Topography not topology
        • Others have examined this approach though mostly in academic research papers
    • Cartographer (II)
      • Model relationships using dependency graph borrowed from graph theory branch of mathematics
      • Systems represented as vertexes
      • Dependencies represented as edges
      • Directed graphs
      • System is server if it provides service to some client
      • System is client if it consumes service
    • Example Dependency Graph
    • What Do We Do With Data?
      • Discover, analyze dependencies
      • Diagnose and troubleshoot faults
      • Security spinoff
      • Monitor, test, & compare service experiences
      • Work bottom-up
    • But I Already Know My Network
      • You may be surprised what you find
      • Distributed systems are highly dynamic, not static
      • Automating management necessitates capturing this info and encoding it
    • What Do We Do? Discover/Analyze
      • Discover dependencies via:
        • OS and app configuration
          • /etc, .ini, and Windows registry
          • System APIs
        • Dynamically via protocol endpoints
          • IPv4 and IPv6
      • Classify into ~ 30 different types
      • Inbound/outbound/transit
      • Per-system, per-user, per-app
    • What We Do? Discover/Analyze (II)
      • Dependencies tell us what a machine is doing
        • Validate configuration and operation
        • Discover misconfiguration
      • Seed automatic configuration for monitoring
        • If DB server => automatically monitor components
    • What Do We Do? Diagnose
      • Who/what is impacted?
        • If key app dies => know who is impacted
      • Determine root cause/impact
        • Given fault, which clients are affected?
        • Given a client, what faults are affecting it?
        • We know service A depends on X,Y,Z
          • If A fails, examine X, Y, Z
    • What Do We Do? Security Spinoff
      • Track dependencies and interactions longterm
      • Develop model of typical behavior/role of system/app
      • Deviations from baseline could indicate issues
      • Social networking for computers
        • If my machine starts communicating with those in China . . . .
    • What Do We Do? Compare Service Experience
      • Do you see what I see?
      • Use dependency data to automatically test services
        • Global, centralized testing
        • Per-system active testing
        • Per-system passive monitoring
      • Detect localized hot-spots
        • Pinpoint infrastructure problems
    • What Is Next Generation About This?
      • Started with observations about how human corporations work
        • CEO sets broad policies and goals
        • Employees implement them, solve problems, run the show
      • Managers and agents become peers
        • Further push intelligence and command/control downward and outward
        • P2P architecture utilized
        • Every agent acts in dual role
    • Peer-to-Peer
      • Not based on polling and storing of data in central repository
        • Not to say this isn't important
      • Agents self-organize into p2p overlay networks
        • Exchange information with peers
        • Run distributed algorithms
        • Self-propagate, self-update
    • What Is A Peer?
      • Systems are peers if they both utilize same service from same server
      • Many p2p overlays
      • Increase scaling (unlimited?)
      • Reduce reaction time
      • Analyze more up-to-date info
    • Example P2P Overlay
    • New Management Framework?
      • Why re-invent the wheel?
        • Could make existing IMF work given enough tape and glue
        • SNMPvX still too cumbersome, inefficient
        • Protocol limitations
        • ASN.1/BER too brittle and prone to interoperability problems
        • WBEM/CIM too heavyweight, complex
          • Spend all day modeling, not managing
        • Some existing work applying XML to IMF
    • X ML M anagement P rotocol
      • Framework in addition to just a protocol
        • SMI, protocol, MIBs
      • Borrow from and extend the IMF as much as possible
      • Utilize XML for:
        • Data modeling (SMI)
        • Specification (MIBs)
        • Transfer syntax (protocol)
      • Everything is text
    • More XML
      • ASN.1 could have been used?
      • More XML tools,
      • More widely adopted than ASN.1
      • XML schemas for structured document
        • Modeling
        • Parsing
        • Conversion
        • Validating
      • Still need to test interoperability
    • XMP SMI
      • http://xmlns.krupczak.org/xsd/xmptypes-1.0.xsd
      • Start with SNMP SMI
      • Enhance only where necessary
        • Do away with OIDs
        • Tuple of MIB-name, object-name, key
        • MIB-2 ifInOctets
          • From:
            • 1.3.6.1.2.1.2.2.1.10.1
          • To:
            • mib2.ifInOctets.if0
    • XMP SMI (II)
      • SMI type enhancements
        • Added several data types and promoted several textual conventions
        • Everything 64-bit min, although with XML, numbers can be larger w/o breaking 2/3 of framework
        • With BER, changing from 32-64 bit breaks SMIs, MIBs, software
        • Textual conventions specify additional semantics; overloading is poor engineering
          • Promote several to standard types
    • XMP SMI (III)
      • Added extendedBoolean type
        • True, False, Unknown
      • Added unsupportedVariable so agent can answer queries honestly and completely
      • Avoid use of inheritance and poloymorphism complexities (ala CIM)
      • Scalar and tabular objects
    • XMP SMI (IV)
      • Tables are relations
        • Support relational table operations
        • How to marry table permissions with object permissions?
      • Need a lot more work on MIB specification & schema
    • XMP Protocol
      • http://xmlns.krupczak.org/xsd/xmp-1.0.xsd
    • XMP Protocol (II)
      • Connection-oriented
        • Avoid much of intricacies of UDP-based protocols
      • What intricacies?
        • More efficient for larger data xfers
        • No need for MIB tricks
        • No need for object ordering
        • No built-in race conditions in large tables
      • Original rationale for SNMP/UDP valid then, not now?
    • XMP Protocol (III)
      • Entity initiates session
      • Also closes session
      • Stay connected as long as needed
      • RPC like semantics
        • Request/response semantics
        • Initiator makes requests
      • Is this a manager?
    • XMP Protocol (IV)
      • Message types borrowed from SNMP
        • GetRequest (scalars)
        • Response (scalars, tables)
        • SetRequest (scalars)
        • Trap
          • First two objects are core.trapType and core.sysObjectID
        • Information
    • Example GetRequest
    • Example Response
    • XMP Table Operations
      • SQL-like
      • SelectTableRequest
      • InsertTableRequest
      • DeleteTableRequest
      • UpdateTableRequest
      • No overloading, no side-effects
    • Example SelectTableRequest
    • No GetNext/GetBulk
      • No GetNext/GetBulk needed for table traversal
      • GetNext yields very little information and no additional semantics
      • But how do I walk a MIB?
        • You don't
        • In practice, walking only yields syntactic information
    • Tables, Keys
      • For scalars, no real instance identifier needed
      • For tables, relation keys
        • Keys can be strings, numbers, variable-length
      • No explicit notion of ordering
        • No need?
    • XMP Encapsulation in SSL/TCP
      • Utilize SSLv3/TLSv1 for privacy and authentication
      • Cartographer utilizes its own CA to create/sign X509v3 certs
      • Each entity embeds own CA
      • Agent -> Agent requires two-way authentication
      • Manager does not need to provide cert
      • TCP/UDP 5270
    • XMP MIBs
      • Virtually compatible with SNMP SMI
      • Implemented MIB-2 in XMP
      • Can implement others
        • HostMIB, SysApplMIB
      • How MIBs are specified still under development
        • XML schema
        • Tables, objects, keys
        • Borrow from relational DB theory and SQL
    • XMP MIBs (II)
      • MIB names must be unique within universe of XMP
      • Within a MIB, object names must be unique
      • Can utilize private-enterprise numbers to help with uniqueness
        • Krupczak.org is 16050
      • Core MIB contains agent-engine stats and config
      • Cartographer MIB implemented
    • But How Do I Make Money?
      • License model:
        • Open source
        • Closed source
        • Dual-license
      • Traditional closed-source company
        • Market for management software mature and consolidating
        • Unlikely to gain much traction
      • Crippleware
    • Example OSS Companies
      • Example open-source companies:
        • Sendmail (OSS, add-on software and services)
        • Snort (dual license?)
        • Asterisk (dual license)
        • OpenNMS (OSS, services)
        • JBoss – sold for $400m to RedHat
        • MySQL – sold for $1B to Sun
    • An Island or Ecosystem?
      • Tremendous investment in existing products & frameworks
      • Add XMP as new management protocol to existing platforms
        • OpenNMS
        • MRTG
        • ZenOSS?
          • Integration in research phase
        • Others?
    • Integration (continued)
      • SNMP/XMP gateway?
        • Not under active consideration
        • Very difficult computer science problem
      • Backport to SNMP, WBEM
        • Not under active consideration
        • More likely than gateway approach
    • Technologies, Platforms, Engines
      • Agent written entirely in C
        • No need to install interpreters, VMs, DLLs
          • In past lifetime, having to install Java on all systems was large barrier
        • Goal is to run agent out of box
        • Very small footprint
          • Footprint less than 3% is upper-bound
        • Engine is 66k lines of C-code
        • Plugins 9k to 16k lines of C-code
      • Ship with libs/DLLs if needed
    • Platform support
      • Solaris 9+ Sparc (64-bit)
      • Solaris 9+ x86
      • Linux 2.4+ on x86 (32, 64-bit)
      • Windows 2000/XP/2003/Vista/2008
        • Win32 and Win64
      • Agent uses as few libs as possible
        • Libxml
        • Pthreads
        • Openssl
        • Iconv, zlib
    • Big Picture
    • Agent Pieces/Parts
    • Licenses
      • Agent engine, GPLv2
      • MIB-2 plugin, GPLv2
      • Example plugin, GPLv2
      • Cartographer plugin, closed source, shrinkwrap software license
      • Java GUI, closed source, shrinkwrap software license
      • See release notes and install instructions
    • Roadmap
      • 1.0 released in November 2008
        • Framework
        • Infrastructure
      • 1.1 release in Spring/Summer 2009
        • Bug fixes, additional platforms
        • MIB schema, SMI work
        • More MIB data
        • More intelligence
        • A lot more work on events
    • More Roadmap
      • 2.0 TBD
        • Self-propagation (already do self-updating)
        • Distributed decision making
        • Root cause, impact
        • Automatic testing/measurement
        • More integration
    • Demo – Cartographer Main
    • Dependency View
    • Dependency Query
    • Dependency Query
    • Dependency Query (Asterisk)
    • Process Query
    • Process Query
    • Endpoint Query
    • Endpoint View
    • MRTG Integration
    • ONMS Integration