Webinar     Toward a Systemic Will to Live –Patterns of Self-Organizing Agile Security                        Rick Dove   ...
AbstractThis talk puts a focus on systems that have an awareness of their environment,and that are sensitive to anomalous ...
General Current Situation       Adversarial Domain (AD)                                  Security Domain (SD)     Adversar...
StaticSystem                                Dynamic                               Adversary          dove@parshift.com,   ...
AsymmetriesAdversary is a natural system, security strategy is an artificial system           Adversary leads with innovat...
Adversarial Advantage                                                   Co-evolution?Architecture:       Multi-agent      ...
Adversarial Advantage                                                            Co-evolution?Architecture:               ...
Mirror the EnemyAgile system security, as a minimum,    must mirror the agile characteristics    exhibited by the system a...
Inspirational Patterns         from natural systems that effectively processnoisy sensory input from uncertain and changin...
Evolution and Innovation                                   Woese, Carl. 2000. Interpreting the universal phylogenetic tree...
www.parshift.com/Files/PsiDocs/PatternQualificationsForAgileSecurity.pdf                     Pattern: Horizontal Gene/Meme...
www.parshift.com/Files/PsiDocs/PatternQualificationsForAgileSecurity.pdf Pattern: Bow Tie Processor (assembler/generator/m...
www.parshift.com/s/110411PatternsForSORNS.pdf                      Pattern: Proactive Anomaly SearchSpeculative generation...
www.parshift.com/s/110411PatternsForSORNS.pdf                    Pattern: Hierarchical Sensemaking      Four level feed fo...
BIS Architecture (Biological Immune System)
Antibody Creation & Life CycleGeneral antibody life cycle: creation, false-positive testing, deployment efficacy or termin...
Antibody Creation & Life CycleGeneral antibody life cycle: creation, false-positive testing, deployment efficacy or termin...
SORNS ApplicationSelf Organizing Resilient Network – Sensing
Proposed Basic Concept Explore advantages of new pattern processor Distribute collaborating detector agents at all netwo...
Reconfigurable Pattern Processor                Reusable Cells Reconfigurable in a Scalable Architecture                  ...
Reconfigurable Pattern Processor                Reusable Cells Reconfigurable in a Scalable Architecture                  ...
Value-Based Feature ExampleA reference pattern example for behavior-verification of a mobile object.         Is it traveli...
SornS Architecture                                               L5 Correlative                                           ...
Detector Creation and Learning Life Cycle                 dove@parshift.com,         24
Endpoint Detector Families – Application Specific                  Connection   SQL Server          Web Server    MS Offic...
Proof-of-ConceptIPv4 packet-header detection– single packet-header signature patterns (spatial connection category)Three e...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Feature Cells and Finite State Machines                     (Illustrative example of pattern processor capability)        ...
Very Large Scale Anomaly Detector
98    126           25   41       250       7                                                                        patte...
Gang Detector (GD)                                               Pattern (or Pattern Path)                                ...
Gang Detector (GD)                                                                                                        ...
Gang Detector (GD)                                                                                                        ...
41Gang Detector (GD)                                                                                         dove@parshift...
42Gang Detector (GD)                                                                                           dove@parshi...
Gang Detector (GD)00 ≈ 001000000000000000                          00 ≈ 010000000100000000                                ...
Gang Detector (GD)                                                                                                        ...
Gang Detector (GD)                                                                                                        ...
Detector Sets                   Create a new           GD                 Gang Detector (GD)                              ...
Detector Sets                   Create a new           GD                   Multiple gang detectors covering              ...
IPv4 Pattern-Space Coverage: c=6          Pattern-Space Coverage by Number of GDs                        % Coverage by num...
Coverage as Function of Cardinality             accelerating decline in coverage                   as cardinality drops,  ...
Coverage of 32 MFDs Declines Fast            6 MFDs at 40% = 98.35% at 1024 GDs                               dove@parshif...
Gang Detector: Some Context   Good for negative selection, not positive selection   You cannot build a GD by adding patt...
SornS Architecture                                               L5 Correlative                                           ...
Toward a Systemic Will to Live - Patters of Self-Organizing Agile Security
Toward a Systemic Will to Live - Patters of Self-Organizing Agile Security
Toward a Systemic Will to Live - Patters of Self-Organizing Agile Security
Toward a Systemic Will to Live - Patters of Self-Organizing Agile Security
Toward a Systemic Will to Live - Patters of Self-Organizing Agile Security
Upcoming SlideShare
Loading in …5
×

Toward a Systemic Will to Live - Patters of Self-Organizing Agile Security

419 views
355 views

Published on

Massive pattern recognition techology, inexpensive, employed as artificial immune system and cortical sense-making for network endpoint self-organizing security - with massive anomally learning and detection capability.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
419
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Toward a Systemic Will to Live - Patters of Self-Organizing Agile Security

  1. 1. Webinar Toward a Systemic Will to Live –Patterns of Self-Organizing Agile Security Rick Dove Last Updated: 8 September 2011 (subject to aperiodic and continuous updates at www.parshift.com/s/TowardsSystemicWillToLive.pdf) Portions of this work were sponsored by the Department of Homeland Security under contract D10PC20039. The content of the material contained herein does not necessarily reflect the position or policy of the Government, and no official endorsement is implied. dove@parshift.com, 1
  2. 2. AbstractThis talk puts a focus on systems that have an awareness of their environment,and that are sensitive to anomalous changes that might signal a threat.Sensitivity to anomalous change is most useful when every possible change,within a domain of interest, accurately triggers attention – meaning no falsepositives (crying wolf) and no false negatives (undetected anomalies).We will first explore four inspirational patterns from natural systems thateffectively process noisy sensory input from uncertain & changing environments: • horizontal gene/meme transfer, • bow tie processors, • proactive anomaly search, and • hierarchical sensemaking.Then the architecture of the biological immune system will be examined, andsubsequently grounded with an artificial immune system example underdevelopment for a resilient cyber-network sense and sensemaking application.Of special note is new anomaly detection technology that enables high fidelityimmune system-like performance, effectively covering a vast detection space of10 to the 15th anomalies in the example shown, with higher capacities practical. dove@parshift.com, 2
  3. 3. General Current Situation Adversarial Domain (AD) Security Domain (SD) Adversarial Security Agent (AA) AA AA Agent (SA) SA SA Adversarial Security Communities AD AD Communities SD SD Dynamic Attack Static ArtifactDynamic attack Static artifacts areincludes human and Relatively Static systems with andsystemic adaptive Security and System without securitycontrol preying upon Artifacts (A) measures, updatedfixed artifact occasionally.defenses. dove@parshift.com, 3
  4. 4. StaticSystem Dynamic Adversary dove@parshift.com, 4
  5. 5. AsymmetriesAdversary is a natural system, security strategy is an artificial system Adversary leads with innovation and evolution Adversary self-organizes as a dynamic system-of-systems … up next … Pattern (Language) ProjectSome Dynamic Self Organizing System-of-System Security Patterns Pattern employment on the SornS project dove@parshift.com, 5
  6. 6. Adversarial Advantage Co-evolution?Architecture: Multi-agent Loosely coupled Self organizing Systems-of-systemsBehavior: Swarm intelligence Tight learning loops Fast evolution Adaptive innovation dove@parshift.com, 6
  7. 7. Adversarial Advantage Co-evolution?Architecture: Not happening. Multi-agent The frog is dragging Loosely coupled us down the block. Self organizing Systems-of-systemsBehavior: We are not in an arms race Swarm intelligence – we haven’t engaged. Tight learning loops Fast evolution Adaptive innovation dove@parshift.com, 7
  8. 8. Mirror the EnemyAgile system security, as a minimum, must mirror the agile characteristics exhibited by the system attack community:[S] Self-organizing – with humans embedded in the loop, or with systemic mechanisms.[A] Adapting to unpredictable situations – with reconfigurable, readily employed resources.[R] Reactively resilient – able to continue, perhaps with reduced functionality, while recovering.[E] Evolving in concert with a changing environment – driven by vigilant awareness and fitness evaluation.[P] Proactively innovative – acting preemptively, perhaps unpredictably, to gain advantage.[H] Harmonious with system functional purpose – aiding rather than degrading system and user productivity. www.parshift.com/Files/PsiDocs/Pap100226-AgileSecuritySelfOrganizingCoEvolution-ExtAbst.pdf dove@parshift.com, 8
  9. 9. Inspirational Patterns from natural systems that effectively processnoisy sensory input from uncertain and changing environments
  10. 10. Evolution and Innovation Woese, Carl. 2000. Interpreting the universal phylogenetic tree. PNAS. 97(15):8392-6. www.ncbi.nlm.nih.gov/pmc/articles/PMC26958/pdf/pq008392.pdfCarl Woese: “Vertically generated and “Vertically generatedhorizontally acquired variation could be variation is necessarilyviewed as the yin and the yang of the highly restricted inevolutionary process. character; it amounts to variations on a lineage’s existing cellular themes. Horizontal transfer, on the other hand, can call on the diversity of the entire biosphere, molecules and systems that have evolvedHorizontal Gene Transfer under all manner of conditions, in a great “HGT” variety of different5 steps leading to the stable inheritance of atransferred gene in a new host. Figure Smets,Barth F. and Tamar Barkay. 2005. Horizontal gene transfer: cellular environments.perspectives at a crossroads of scientific disciplines.Nature Reviews Microbiology 3, 675-678 (Sep 2005). Thus, horizontally“The vast majority, between 88% and 98%, of the expansions of protein families [in eight studied derived variation is theprokaryote clades] are due to HGT. … Xenologs [external transfers] have an average age ofintroduction that is twice that of paralogs [internal transfers]. Xenologs are therefore more major, if not the sole,persistent.” Treangen, Todd J. and Eduardo P. C. Rocha. 2011. Horizontal Transfer, Not evolutionary source ofDuplication, Drives the Expansion of Protein Families in Prokaryotes. PLoS Genetics 7:1, January. dove@parshift.com, true innovation.” 10
  11. 11. www.parshift.com/Files/PsiDocs/PatternQualificationsForAgileSecurity.pdf Pattern: Horizontal Gene/Meme Transfer Intrachromsomal genes Extrachromosomal genes Rules 1. Packaging 2. Transfer 3. Entry Available high variety Two modular 4. Establishment Innovative adaptation cellular organisms gene pools 5. Inheritance and evolution Horizontal gene transfer speeds up innovative short-term adaptation and long-term evolutionContext: When conditions deteriorate, it makes a lot of sense to try to scavenge DNA fromyour neighbors. Horizontal gene transfer facilitates a fast microbial adaptation to stress.Higher-than-suspected transfer rates among microbes living in nutrient-poor environments,where sharing genes may be key to survival, has been observed. Evidence indicates thatorganisms limit gene exchange to microbes on nearby branches of the family tree, probablybecause their chromosomes share certain characteristics. Genes appear to be exchangedbetween species with similar chromosomal structures (Pennise 2011).Problem: Situational or environmental changes that threaten fitness or survival of theorganism.Forces: Short-term adaptability vs. long-term-evolvability, horizontal gene transfer speedsthe development of new traits by a factor of 10,000 (Woese 2000, Pennise 2011).Solution: Incorporate appropriate genetic material from other organisms that havedeveloped compatible and useful situational fitness. Mobile genes don’t just help acommunity survive, they also provide the grist for evolutionary innovations. dove@parshift.com, 11
  12. 12. www.parshift.com/Files/PsiDocs/PatternQualificationsForAgileSecurity.pdf Pattern: Bow Tie Processor (assembler/generator/mediator) V: 123 Variable segments V1 123 Vs Vn ~106 VDJ+VJ possible antigen detector shapes D1 27 Ds Dn D: 27 Diverse segments J1 6 Js Jn increases to ~109 varieties with J: 6 Joining segments addition of random nucleotide connections Vr r Dr r Jr 1 random from each + random connect between VDJ & VJ joinings Available high variety Evolve three fixed V-D-J Fixed-rule VDJ assembly Random high variety output genetic DNA input gene-segment libraries with random interconnects with VDJ + VJ assemblies Millions of random infection detectors generated continuously by fixed rules and modules in the “knot”Context: Complex system with many diverse inputs and many diverse outputs, whereoutputs need to respond to many needs or innovate for many or unknown opportunities, andit is not practical to build unique one-to-one connections between inputs and outputs.Appropriate examples include common financial currencies that mediate between producersand consumers, the adaptable biological immune system that produces proactive infectiondetectors from a wealth of genetic material, and the Internet protocol stack that connectsdiverse message sources to diverse message sinks.Problem: Too many connection possibilities between available inputs and useful outputs tobuild unique robust, evolving satisfaction-processes between each.Forces: Large knot short-term-flexibility vs small knot short-term-controllability and long-term-evolvability (Csete 2004); robustness to known vs fragility to unknown (Carlson 2002).Solution: Construct relatively small “knot” of fixed modules from selected inputs, that canbe assembled into outputs as needed according to a fixed protocol. A proactive example isthe adaptable immune system that constructs large quantities of random detectors(antigens) for unknown attacks and infections. A reactive example is a manufacturing linethat constructs products for customers demanding custom capabilities. dove@parshift.com, 12
  13. 13. www.parshift.com/s/110411PatternsForSORNS.pdf Pattern: Proactive Anomaly SearchSpeculative generation and mutation of detectors recognizes new attacks like a biological immune systemContext: A complex system or system-of-systems subject to attack and infection, with lowtolerance for attack success and no tolerance for catastrophic infection success; withresilient remedial action capability when infection is detected. Appropriate examples includebiological organisms, and cyber networks for military tactical operations, national criticalinfrastructure, and commercial economic competition.Problem: Directed attack and infection types that constantly evolve in new innovative waysto circumvent in-place attack and infection detectors.Forces: False positive tradeoffs with false negatives, system functionality vs functionalityimpairing detection measures, detectors for anything possible vs added costs ofcomprehensive detection, comprehensive detection of attack vs cost of false detection ofself.Solution: A high fidelity model of biological immune system antibody (detection) processesthat generate high quantity and variety of anticipatory speculative detectors in advance ofattack and during infection, and evolve a growing memory of successful detectors specificto the nature of the system-of-interest. dove@parshift.com, 13
  14. 14. www.parshift.com/s/110411PatternsForSORNS.pdf Pattern: Hierarchical Sensemaking Four level feed forward/backward sense-making hierarchy modeled on visual cortexContext: A decision maker in need of accurate situational awareness in a critical dynamicenvironment. Examples include a network system administrator in monitoring mode andunder attack, a military tactical commander in battle, and the NASA launch control room.Problem: A very large amount of low-level noisy sensory data overwhelms attempts toexamine and conclude what relevance may be present, most especially if time is importantor if sensory data is dynamic.Forces: amount of data to be examined vs time to reach a conclusion, number of ways datacan be combined vs number of conclusions data can indicate, static sensory data vsdynamic sensory data, noise tolerated in sensory data vs cost of low noise sensory data.Solution: Using a bow-tie process, each level looks for a specific finite set of data patternsamong the infinite possibilities of its input combinations, aggregating its input data intospecific chunks of information. These chunks are fed-forward to the next higher level, thattreats them in turn as data further aggregated into higher forms of information chunks.Through feedback, a higher level may bias a lower level to favor certain chunks over others,predicting what is expected now or next according to an emerging pattern at the higher level.Each level is only interested in a small number of an infinite set of data-combinationpossibilities, but as aggregation proceeds through multiple levels, complex dataabstractions and recognitions are enabled.dove@parshift.com, 14
  15. 15. BIS Architecture (Biological Immune System)
  16. 16. Antibody Creation & Life CycleGeneral antibody life cycle: creation, false-positive testing, deployment efficacy or termination, mutationimprovement, and long-term memory.1. Candidate antibody semi-randomly created.2. Tolerization period tests immature candidates for false-positive matches.3. Mature & naïve antibodies put into time limited service.4. Activated (B-cell) antibodies need co-stimulation (by T-cells) to ensure “improvement” didn’t produce auto-reactive result, non-activated & non-co-stimulated candidates die when time limit ends. 1 5. Highest affinity co-stimulated antibodies are remembered for time-limited long term (eg, many years, decades). 6. Co-stimulated antibodies are cloned with structured mutations, looking for improved (higher) affinity scores. 2 3 4 6 5Diagram modified from (Hofmeyr 2000). dove@parshift.com, 16
  17. 17. Antibody Creation & Life CycleGeneral antibody life cycle: creation, false-positive testing, deployment efficacy or termination, mutationimprovement, and long-term memory.1. Candidate antibody semi-randomly created.2. Tolerization period tests immature candidates for false-positive matches.3. Mature & naïve antibodies put into time limited service.4. Activated (B-cell) antibodies need co-stimulation (by T-cells) to ensure “improvement” didn’t produce auto-reactive result, non-activated & non-co-stimulated candidates die when time limit ends. 1 5. Highest affinity co-stimulated antibodies are remembered for time-limited long term (eg, many years, decades). 6. Co-stimulated antibodies are cloned with structured mutations, looking for improved (higher) affinity scores. 2 Shape/Pattern Space ~109 3 4 6 5 Self nonself discrimination: A universe of data points is partitioned into two sets – self and nonself. Negative detectors cover subsets of non-self. From (Esponda 2004)Diagram modified from (Hofmeyr 2000). dove@parshift.com, 17
  18. 18. SORNS ApplicationSelf Organizing Resilient Network – Sensing
  19. 19. Proposed Basic Concept Explore advantages of new pattern processor Distribute collaborating detector agents at all network endpoints Artificial Immune System pattern detection a la Forest/Hofmeyr, et al. Hierarchal sensemaking a la Fink/Fulp, Hawkins/George, et al. Implement the work of others in a less-constraining technology that can better approach high fidelity natural-system performance dove@parshift.com, 19
  20. 20. Reconfigurable Pattern Processor Reusable Cells Reconfigurable in a Scalable Architecture www.parshift.com/Files/PsiDocs/Pap090303-PatternRecognitionWithoutTradeoffs.pdf Independent detection cell: Cell-satisfaction content addressable output pointers by current input byte Up to 256 possible features If active, and satisfied with can be “satisfied” by all current byte, can activate so-designated byte values other designated cells including itself Cell-satisfaction activation pointers Individual detection cells are configured into detectors by linking activation pointers.an unbounded number of detector cells configured as finite state machines can extend indefinitely across multiple processors All active cells have simultaneous access to current data-stream byte dove@parshift.com, 20
  21. 21. Reconfigurable Pattern Processor Reusable Cells Reconfigurable in a Scalable Architecture www.parshift.com/Files/PsiDocs/Pap090303-PatternRecognitionWithoutTradeoffs.pdf Independent detection cell: Cell-satisfaction content addressable output pointers by current input byte Up to 256 possible features If active, and satisfied with can be “satisfied” by all current byte, can activate so-designated byte values other designated cells including itself Cell-satisfaction activation pointers Individual detection cells are configured into detectors by linking activation pointers. Enables High Fidelity Modelingan unbounded number of detector cells configured as finite state machines can extend indefinitely across multiple processors All active cells have simultaneous access to current data-stream byte dove@parshift.com, 21
  22. 22. Value-Based Feature ExampleA reference pattern example for behavior-verification of a mobile object. Is it traveling within the planned space/time envelop? Using GPS position data: Latitude, Longitude, Altitude. linear, log or other scale F F S Output absolute relative F = failure S = success 256 distance values minimum separation L L A L L A L L A A O L A O L A O L T N T T N T T N T FCM configured to showing acceptable ranges of values classify failure/success Paths and Methods For Peer Behavior Monitoring Among Unmanned Autonomous Systems, www.parshift.com/Files/PsiDocs/Paths&MethodsForPeerBehaviorMonitoringAmongUnmannedAutonomousSystems.pdf dove@parshift.com, 22
  23. 23. SornS Architecture L5 Correlative Detection Self Organizing Resilient Network Agent/Human Sensing (& Sensemaking) Architecture anticipates collaboration Policy/Procedure among SORN networks by L4 agents Interface Data Base Network Network L4 Correlative Detection Agent // L4 Correlative Detection Agent SORNS Hardware Device End Point End Point L3 Correlative Detection Agent // L3 Correlative Detection Agent L2 Temporal Detection Agent // L2 Temporal Detection Agent Phase 1 Focus L1 Spatial Detector Agent // L1 Spatial Detector Agent noisy sensor stream noisy sensor stream (e.g., packets, log files) (e.g., packets, log files) Multi-level hierarchy refines situational awareness with learning and sensemaking, supports remedial action agents (human/automated) with succinct relevant information.Notes:• For general collaborative hierarchy concept see (Haack 2009)• For hierarchical feed-forward/backward pattern learning, prediction, and sense-making see (George 2009).• For hierarchical learning of causal patterns spread as time-sequence events see (Hawkins 2010, Hawkins et al 2010). dove@parshift.com, 23
  24. 24. Detector Creation and Learning Life Cycle dove@parshift.com, 24
  25. 25. Endpoint Detector Families – Application Specific Connection SQL Server Web Server MS Office …Appn Spatial Detector Detector Detector Detector Detector Family Family Family Family FamilyTemporal Detector Detector Detector Detector Detector Family Family Family Family FamilyCorrelative Detector Detector Detector Detector Detector Family Family Family Family Family Detection philosophy: Automated learning of pattern features within a fixed set of generic pattern structures. Spatial: n specific things happened in contiguous order Temporal: n specific things happened in order Correlative: n specific things happened dove@parshift.com, 25
  26. 26. Proof-of-ConceptIPv4 packet-header detection– single packet-header signature patterns (spatial connection category)Three elements to a pattern signature: address – port – type• Address: 4 bytes - Only the non-host address is of interest.• Port: 2 bytes - Only the destination port is of interest.• Types: 3 bits covers 8 types – (TCP, UDP, ICMP, other) x (incoming, outgoing)The L1-Agent preprocessor/controller selects relevant features from networkpackets and feeds them as condensed “feature packets” to the pattern processor L2 Agent L1 Agent IPv4 Pattern Processor network • conventional processor/memory feature packets • special purpose chip packets • detector generator • detectors in nursery • feature packet assembly detection • detectors in service • pattern processor controller alert • detectors in memory dove@parshift.com, 26
  27. 27. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○○○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value.If the index findsa set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 27
  28. 28. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○○○○○○○○○○○○●○○ ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Loaded with 7 valuesIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. 192.168.1.44 0.118 2 dove@parshift.com, 28
  29. 29. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○ ○○ ≈ ○○○○○○○○○○○○○○○●○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Processing Data StreamIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 29
  30. 30. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○ ○○ ≈ ○○○○○○○○○○○○○○○●○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Processing Data StreamIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 30
  31. 31. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○ ○○ ≈ ○○○○○○○○○○○○○○○●○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Processing Data StreamIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 31
  32. 32. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○ ○○ ≈ ○○○○○○○○○○○○○○○●○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Processing Data StreamIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 32
  33. 33. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○ ○○ ≈ ○○○○○○○○○○○○○○○●○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Processing Data StreamIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 33
  34. 34. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○ ○○ ≈ ○○○○○○○○○○○○○○○●○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Processing Data StreamIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 34
  35. 35. Feature Cells and Finite State Machines (Illustrative example of pattern processor capability) 7 multi-feature detectors “connected” as a finite state machine (FSM) start end256-bitassociative ○○ ≈ ○○○○●○○○○○○○○○○○○○ ○○ ≈ ○○○○○○○○○●○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○●○ ○○ ≈ ○○○○○○○○●○○○○○○○○○ ○○ ≈ ○○○○○○○○○○○○○○○○○● ○○ ≈ ○○○○○○○○○○○○○●○○○○ ○○ ≈ ○○○○○○○○○○○○○○○●○○memorymulti-featuredetectors (MFD).All active MFDsare indexed bythe inputstream’s currentbyte value. Processing Data StreamIf the index finds 192.168.1.44, 0.118, 2a set bit, thenext MFD isactivated andlooks at the nextstream byte,else the process IPv4 address port typedies. dove@parshift.com, 35
  36. 36. Very Large Scale Anomaly Detector
  37. 37. 98 126 25 41 250 7 pattern Fundamental 98 126 25 41 11 0 Elementspattern list 255 255 255 255 255 255 16 13 123 255 0 0 255 255 255 255 255 254 255 255 255 255 255 253 255 255 255 255 255 0 pattern 255 255 255 255 254 255 pattern space contains space 2566 = 2.81x1014 255 255 255 255 254 254 255 255 255 255 254 253 unique patterns 255 255 255 255 254 0 for patterns consisting of 6 feature values with range 0-255 0 0 0 0 0 0 feature packet feature value 12 98 126 25 41 250 7 16 13 123 255 0 0 98 1 feature stream dove@parshift.com, 37
  38. 38. Gang Detector (GD) Pattern (or Pattern Path) Feature Indicator10 ≈ 001001110010101111 01 ≈ 011111110110101001 11 ≈ 110011100011101111 10 ≈ 001100110110111100 00 ≈ 001111110010101011 11 ≈ 001010110100101010 (1-bit) Non-Feature Indicator (0-bit) Multi-Feature Detectors (MFD, variable size) 10101111 Gang Detector (eg, with seven multi-feature detectors) dove@parshift.com, 38
  39. 39. Gang Detector (GD) A GD is implemented as a 2 dimensional bit array, with each column corresponding to an MFD of independent size, but typically a max Pattern (or Pattern Path) of 256 to accommodate associative addressing (indexing) by an 8-bit Feature Indicator Feature Packet byte.10 ≈ 001001110010101111 01 ≈ 011111110110101001 11 ≈ 110011100011101111 10 ≈ 001100110110111100 00 ≈ 001111110010101011 11 ≈ 001010110100101010 (1-bit) A Feature Indicator is a 1-bit in any or all of the possible index values. Non-Feature Indicator (0-bit) One GD with all Feature Indicators present would have Multi-Feature Detectors 256x256x256x256x256x256x8 = (MFD, variable size) 2.6x1015 unique Pattern Paths. 10101111 This many unique patterns would be represented in just (6x32)+1 = 193 8-bit data bytes. Gang Detector If each of these patterns were in a (eg, with seven multi-feature detectors) pattern list, seven times the number of possible patterns in data bytes would be required = ~1016 data bytes in contrast. dove@parshift.com, 39
  40. 40. Gang Detector (GD) A GD is implemented as a 2 dimensional bit array, with each column corresponding to an MFD of independent size, but typically a max Pattern (or Pattern Path) of 256 to accommodate associative addressing (indexing) by an 8-bit Feature Indicator Feature Packet byte.10 ≈ 001001110010101111 01 ≈ 011111110110101001 11 ≈ 110011100011101111 10 ≈ 001100110110111100 00 ≈ 001111110010101011 11 ≈ 001010110100101010 (1-bit) A Feature Indicator is a 1-bit in any or all of the possible index values. Non-Feature Indicator (0-bit) One GD with all Feature Indicators present would have Multi-Feature Detectors 256x256x256x256x256x256x8 = (MFD, variable size) 2.6x1015 unique Pattern Paths. 10101111 This many unique patterns would be represented in just (6x32)+1 = 193 8-bit data bytes. Gang Detector If each of these patterns were in a (eg, with seven multi-feature detectors) pattern list, seven times the number of possible patterns in data bytes would be required = ~1016 data bytes in contrast. dove@parshift.com, a unique benefit of the approach 40
  41. 41. 41Gang Detector (GD) dove@parshift.com, 7 Feature Indicators = 1 Pattern (Path) 00001000 00 ≈ 000000000000001000 00 ≈ 000000000000100000 00 ≈ 000000000000100000 00 ≈ 000001000000000000 00 ≈ 000000000100000000 00 ≈ 001000000000000000
  42. 42. 42Gang Detector (GD) dove@parshift.com, 8 Feature Indicators = 2 Patterns (Paths) 7 Feature Indicators = 1 Pattern (Path) 00001000 00 ≈ 000000000000001000 00 ≈ 000000000000100000 00 ≈ 000000000000100000 00 ≈ 000001000000000000 00 ≈ 010000000100000000 00 ≈ 001000000000000000
  43. 43. Gang Detector (GD)00 ≈ 001000000000000000 00 ≈ 010000000100000000 00 ≈ 000001000000000000 00 ≈ 000000100000100000 00 ≈ 000000000000100000 00 ≈ 000000000000001000 00001000 7 Feature Indicators = 1 Pattern (Path) 8 Feature Indicators = 2 Patterns (Paths) 9 Feature Indicators = 4 Patterns (Paths) dove@parshift.com, 43
  44. 44. Gang Detector (GD) Adding a single Feature Indicator increases the Patterns (Paths) by a factor of 2, an exponential increase.00 ≈ 001000000000000000 00 ≈ 010000000100000000 00 ≈ 000001000000000000 00 ≈ 000000100000100000 00 ≈ 000000000000100000 00 ≈ 000000000100001000 An application might create a new GD with the same percentage of Feature Indicators in every MFD. If that were 50%, with six MFDs of size 256 and one of size 8, the total number of Patterns (Paths) upon creation would be 00001000 128x128x128x128x128x128x4= 1.8x1013 patterns Detectable at data-stream feed-speed 7 Feature Indicators = 1 Pattern (Path) independent of the number of patterns 8 Feature Indicators = 2 Patterns (Paths) 9 Feature Indicators = 4 Patterns (Paths) 10 Feature Indicators = 8 Patterns (Paths) dove@parshift.com, 44
  45. 45. Gang Detector (GD) Adding a single Feature Indicator increases the Patterns (Paths) by a factor of 2, an exponential increase.00 ≈ 001000000000000000 00 ≈ 010000000100000000 00 ≈ 000001000000000000 00 ≈ 000000100000100000 00 ≈ 000000000000100000 00 ≈ 000000000100001000 An application might create a new GD with the same percentage of random Feature Indicators in every MFD. If that were 50%, with six MFDs of size 256 and one of size 8, the total number of Patterns (Paths) upon creation would be 00001000 128x128x128x128x128x128x4= 1.8x1013 patterns Detectable at data-stream feed-speed 7 Feature Indicators = 1 Pattern (Path) independent of the number of patterns 8 Feature Indicators = 2 Patterns (Paths) 9 Feature Indicators = 4 Patterns (Paths) 10 Feature Indicators = 8 Patterns (Paths) a unique benefit of the approach dove@parshift.com, 45
  46. 46. Detector Sets Create a new GD Gang Detector (GD) Creation Mature new GD GD in the Nursery Maturation Insert mature GD GD into Service Insertion GD Use GDs to detect anomaliesDetection GD Remove GDs from ServiceRemoval GDN1 GDN2 GDNn GDS1 GDS2 GDSm DM1 DM2 DMm DA1 DA2 DAa Nursery Set: Service Set: Memory Set: Action Set: mass patterns mass patterns single/multi patterns single patterns dove@parshift.com, 46
  47. 47. Detector Sets Create a new GD Multiple gang detectors covering Gang Detector (GD) Creation slightly-overlapping portions of total pattern space collectively increase the Mature new GD GD total coverage of pattern space. in the Nursery Maturation Insert mature GD GD 50% of Feature Indicators set across 7 MFDs (6@256 & 1@8) into Service Insertion GD Use GDs to detect anomaliesDetection GD Remove GDs 99.97% coverage from ServiceRemoval with 512 GDs GDN1 GDN2 GDNn GDS1 GDS2 GDSm DM1 DM2 DMm DA1 DA2 DAa Nursery Set: Service Set: Memory Set: Action Set: mass patterns mass patterns single/multi patterns single patterns dove@parshift.com, 47
  48. 48. IPv4 Pattern-Space Coverage: c=6 Pattern-Space Coverage by Number of GDs % Coverage by number of GDs100.00%90.00% 99.35% 99.76% 99.91% 99.97% 100.00% 95.14% 98.23%80.00% 86.68% 90.00%70.00% 80.00%60.00% 70.00% 63.50%50.00% 60.00%40.00% 50% cardinality 50.00%30.00% 40.00%20.00% 30.00% 20.00%10.00% 10.00% 0.00% 0.00% 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321 341 361 381 401 421 441 461 481 501 64 128 192 256 320 384 448 512 IPv6 Pattern-Space Coverage: c=32 Pattern-Space Coverage by Number of GDs % Coverage by number of GDs100.00% 100.00% 95.03%90.00% 89.48% 92.77%80.00% 90.00% 84.69% 77.71%70.00% 80.00% 67.56%60.00% 70.00%50.00% 60.00% 52.79%40.00% 85% cardinality 50.00% 40.00% 31.29%30.00% 30.00%20.00% 20.00%10.00% 10.00% 0.00% 0.00% 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321 341 361 381 401 421 441 461 481 501 64 128 192 256 320 384 448 512 dove@parshift.com, 48
  49. 49. Coverage as Function of Cardinality accelerating decline in coverage as cardinality drops, 40% thought comfortable thresholdCardinality losses justify the value of refresh-cycling the in-service GDs, and sharing results with other endpoint agents dove@parshift.com, 49
  50. 50. Coverage of 32 MFDs Declines Fast 6 MFDs at 40% = 98.35% at 1024 GDs dove@parshift.com, 50
  51. 51. Gang Detector: Some Context Good for negative selection, not positive selection You cannot build a GD by adding patterns to it (in general) You cannot delete a single pattern from it (like Bloom Filters that way) 00 ≈ 001000000000000000 00 ≈ 010000000100001000 00 ≈ 000001000000000000 00 ≈ 000000000000100000 00 ≈ 000000000000100000 00 ≈ 000000000000001000 The exception, multi- patterns can be built with a single MFD 00001000 3 Pattern Paths with no spurious paths introduced dove@parshift.com, 51
  52. 52. SornS Architecture L5 Correlative Detection Self Organizing Resilient Network Agent/Human Sensing (& Sensemaking) Architecture anticipates collaboration Policy/Procedure among SORN networks by L4 agents Interface Data Base Network Network L4 Correlative Detection Agent // L4 Correlative Detection Agent SORNS Hardware Device End Point End Point L3 Correlative Detection Agent // L3 Correlative Detection Agent L2 Temporal Detection Agent // L2 Temporal Detection Agent Phase 1 Focus L1 Spatial Detector Agent // L1 Spatial Detector Agent noisy sensor stream noisy sensor stream (e.g., packets, log files) (e.g., packets, log files) Multi-level hierarchy refines situational awareness with learning and sensemaking, supports remedial action agents (human/automated) with succinct relevant information.Notes:• For general collaborative hierarchy concept see (Haack 2009)• For hierarchical feed-forward/backward pattern learning, prediction, and sense-making see (George 2009).• For hierarchical learning of causal patterns spread as time-sequence events see (Hawkins 2010, Hawkins et al 2010). dove@parshift.com, 52

×