Modeling IT Infrastucture
using
The Assimilation Project
#AssimProj

@OSSAlanR

http://assimproj.org/
http://bit.ly/AssimG...
Upcoming Events
GraphConnect San Francisco (today!)
Open Source Monitoring Conference - Nürnberg
NSA / Homeland Security A...
Talk Outline
●

Project Overview

●

Infrastructure Schema

●

Python Object-Graph-Mapping API

G
R
A
P
H
C
O
N
N
E
C
T

G...
Assimilation Project History
●

Inspired by 2 million core computer (cyclops64)

●

Concerns for extreme scale

●

Topolog...
Project Scope
Zero-network-footprint continuous Discovery
integrated with extreme-scale Monitoring
●

Continuous extensibl...
Discovery
Discovering
●

systems you've forgotten

●

what you're not monitoring

●

whatever you'd like

●

without setti...
Why Discovery?
●

Documentation: incomplete, incorrect

●

Dependencies: unknown

●

Planning: Needs accurate data

●

●

...
Why Neo4j (graph db)?
●
●

●

●
●

●

Dependency & Discovery information: graph
Speed of graph traversals depends on size
...
Assimilation Communication
Neighbor-Rings

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013
© 2013 Assimilation System...
Ring Representation Schema

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited

10/2...
ssh -> sshd dependency graph

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited

11...
Switch Discovery Data
from LLDP (or CDP)

CRM transforms LLDP (CDP) Data to JSON
© 2013 Assimilation Systems Limited

G
R
...
OGM – Object Graph Mapping
●

Managing the Graph Nodes “disappears”

●

The Object Model is the Graph Model

●

Significan...
OGM rules
●
●

●

Don't use Constructors
Relationships replace hash tables and
object references and so on
Constructor par...
OGM sample
@RegisterGraphClass
class Person(GraphNode):
'A Person - or at least an electronic representation of one'
def _...
OGM sample
@RegisterGraphClass
class TestSystem(GraphNode):
'Some kind of semi-intelligent system'
def __init__(self, desi...
OGM sample
@staticmethod
def __meta_keyattrs__():
'Return our key attributes in order of significance'
return ['designatio...
OGM sample
# (seven)-[:formerly]->(Annika)
Annika = store.load_or_create(Person,
firstname='Annika',
lastname='Hansen')
se...
OGM sample
@RegisterGraphClass
class TestDrone(TestSystem):
def __init__(self, designation, roles=[]):
TestSystem.__init__...
Current State
●

First release was April 2013

●

Great unit test infrastructure

●

Nanoprobe code – works well
●

Servic...
Future Plans
●

Production grade by end of year

●

Purchased support

●

“Real digital signatures, compression, encryptio...
Get Involved!
Powerful Ideas and Infrastucture
Fun, ground-breaking project
Looking for early adopters, testers!!
Needs fo...
Resistance Is Futile!
#AssimProj

@OSSAlanR

#AssimMon

Project Web Site
http://assimproj.org

G
R
A
P
H
C
O
N
N
E
C
T

Bl...
The Elder GeekGirl

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited

24/25
Younger GeekGirl's Computer

Running Linux
Of Course!

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013
© 2013 Assimil...
Upcoming SlideShare
Loading in …5
×

Graphing Enterprise IT – Representing IT Infrastructure and Business Processes as a Graph - Alan Robertson @ GraphConnect SF 2013

6,769 views
6,647 views

Published on

The Assimilation Monitoring Project is an open source project providing both continuous Stealth Discovery(TM) and extreme scale monitoring.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,769
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
16
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Graphing Enterprise IT – Representing IT Infrastructure and Business Processes as a Graph - Alan Robertson @ GraphConnect SF 2013

  1. 1. Modeling IT Infrastucture using The Assimilation Project #AssimProj @OSSAlanR http://assimproj.org/ http://bit.ly/AssimGC2013 Alan Robertson <alanr@unix.sh> Assimilation Systems Limited http://assimilationsystems.com G R A P H C O N N E C T
  2. 2. Upcoming Events GraphConnect San Francisco (today!) Open Source Monitoring Conference - Nürnberg NSA / Homeland Security Assimilation Technical Talk Large Installation System Administration Conference - DC Colorado Springs Open Source User’s Group linux.conf.au – Awesome Australian Linux Conf - Perth Details on http://assimilationsystems.com/ G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 2/25
  3. 3. Talk Outline ● Project Overview ● Infrastructure Schema ● Python Object-Graph-Mapping API G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 3/25
  4. 4. Assimilation Project History ● Inspired by 2 million core computer (cyclops64) ● Concerns for extreme scale ● Topology aware monitoring ● Topology discovery w/out security issues =►Discovery of everything! G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 4/25
  5. 5. Project Scope Zero-network-footprint continuous Discovery integrated with extreme-scale Monitoring ● Continuous extensible discovery – ● Extensible exception monitoring – ● systems, switches, services, dependencies – zero network footprint G R A P H C O N N E C T more than 100K systems All data goes into central graph database © 2013 Assimilation Systems Limited GraphConnect 4 October 2013 5/25
  6. 6. Discovery Discovering ● systems you've forgotten ● what you're not monitoring ● whatever you'd like ● without setting off security alarms G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 6/25
  7. 7. Why Discovery? ● Documentation: incomplete, incorrect ● Dependencies: unknown ● Planning: Needs accurate data ● ● Best Practices: Verification needs data ITIL CMDB (Configuration Mgmt DataBase) Our Discovery: continuous, low-profile © 2013 Assimilation Systems Limited G R A P H C O N N E C T GraphConnect 4 October 2013 7/25
  8. 8. Why Neo4j (graph db)? ● ● ● ● ● ● Dependency & Discovery information: graph Speed of graph traversals depends on size of subgraph, not total graph size Root cause queries  graph traversals – notoriously slow in relational databases Visualization of relationships Schema-less design: good for constantly changing heterogeneous environment Graph Model === Object Model © 2013 Assimilation Systems Limited G R A P H C O N N E C T GraphConnect 4 October 2013 8/25
  9. 9. Assimilation Communication Neighbor-Rings G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 9/25
  10. 10. Ring Representation Schema G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 10/25
  11. 11. ssh -> sshd dependency graph G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 11/25
  12. 12. Switch Discovery Data from LLDP (or CDP) CRM transforms LLDP (CDP) Data to JSON © 2013 Assimilation Systems Limited G R A P H C O N N E C T GraphConnect 4 October 2013 12/25
  13. 13. OGM – Object Graph Mapping ● Managing the Graph Nodes “disappears” ● The Object Model is the Graph Model ● Significant Improvement in Thinking Clarity – The “mud” is gone G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 13/25
  14. 14. OGM rules ● ● ● Don't use Constructors Relationships replace hash tables and object references and so on Constructor parameter names match attribute names G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 14/25
  15. 15. OGM sample @RegisterGraphClass class Person(GraphNode): 'A Person - or at least an electronic representation of one' def __init__(self, firstname, lastname, dateofbirth=None): GraphNode.__init__(self, domain='global') self.firstname = firstname self.lastname = lastname if dateofbirth is not None: self.dateofbirth = dateofbirth else: self.dateofbirth='unknown' @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['lastname', 'firstname'] © 2013 Assimilation Systems Limited G R A P H C O N N E C T GraphConnect 4 October 2013 15/25
  16. 16. OGM sample @RegisterGraphClass class TestSystem(GraphNode): 'Some kind of semi-intelligent system' def __init__(self, designation, roles=[]): GraphNode.__init__(self, domain) self.designation = designation.lower() if roles == []: roles = [''] self.roles = roles def addroles(self, role): if self.roles[0] == '''': del self.roles[0] if isinstance(role, list): for arole in role: self.addroles(arole) elif role not in self.roles: self.roles.append(role) © 2013 Assimilation Systems Limited G R A P H C O N N E C T GraphConnect 4 October 2013 16/25
  17. 17. OGM sample @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['designation'] G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 17/25
  18. 18. OGM sample # (seven)-[:formerly]->(Annika) Annika = store.load_or_create(Person, firstname='Annika', lastname='Hansen') seven = store.load_or_create(Drone, designation='SevenOfNine', roles='Borg') store.relate(seven, 'formerly', Annika) store.commit() G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 18/25
  19. 19. OGM sample @RegisterGraphClass class TestDrone(TestSystem): def __init__(self, designation, roles=[]): TestSystem.__init__(self, designation=designation) if isinstance(roles, str): roles = [roles] roles.extend(['host', 'Drone']) System.__init__(self, designation, roles=roles) G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 19/25
  20. 20. Current State ● First release was April 2013 ● Great unit test infrastructure ● Nanoprobe code – works well ● Service monitoring works ● Lacks digital signatures, encryption, compression ● Reliable UDP comm code working ● Several discovery methods written ● CMA and database code restructuring near-complete ● G R A P H C O N N E C T UI development underway ● Licensed under the GPL, commercial license available © 2013 Assimilation Systems Limited GraphConnect 4 October 2013 20/25
  21. 21. Future Plans ● Production grade by end of year ● Purchased support ● “Real digital signatures, compression, encryption ● Other security enhancements ● Much more discovery ● GUI ● Alerting ● Reporting ● Add Statistical Monitoring ● Best Practice Audits ● Dynamic (aka cloud) specialization ● G R A P H C O N N E C T Hundreds more ideas – See: https://trello.com/b/OpaED3AT © 2013 Assimilation Systems Limited GraphConnect 4 October 2013 21/25
  22. 22. Get Involved! Powerful Ideas and Infrastucture Fun, ground-breaking project Looking for early adopters, testers!! Needs for every kind of skill ● ● ● ● ● ● ● Awesome User Interfaces (UI/UX) Evangelism, community building Test Code (simulate 106 servers!) Python, C, script coding Documentation Feedback: Testing, Ideas, Plans Many others! © 2013 Assimilation Systems Limited G R A P H C O N N E C T GraphConnect 4 October 2013 22/25
  23. 23. Resistance Is Futile! #AssimProj @OSSAlanR #AssimMon Project Web Site http://assimproj.org G R A P H C O N N E C T Blog techthoughts.typepad.com lists.community.tummy.com/cgi-bin/mailman/admin/assimilation © 2013 Assimilation Systems Limited GraphConnect 4 October 2013 23/25
  24. 24. The Elder GeekGirl G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 24/25
  25. 25. Younger GeekGirl's Computer Running Linux Of Course! G R A P H C O N N E C T GraphConnect 4 October 2013 © 2013 Assimilation Systems Limited 25/25

×