Near Real-time Outlier Detection and Interpretation - Part 1 by Robert Thorman, AT&T
•Download as PPTX, PDF•
0 likes•313 views
With the advent of Big Data in the Threat Analytics space needs emerge to perform near real-time (NRT) threat detection and automated interpretation that speed counter measures and remediation. AT&T Chief Security Organization (CSO) has developed an enterprise architecture that includes near real-time outlier processes necessary to protect its network from cyber threats using the Hadoop ecosystem. One enterprise challenge that CSO has faced is summarized in the statement by Brian Rexroad, Executive Director of Technology and Security: "I feel there is too much emphasis is on "detecting". Significantly more emphasis is needed in automated extraction of related information/activity and interpretation of that information." Therefore; CSO Engineering team developed the Stratum™ architecture that includes many open source and commercial products facilitating the rapid development and operationalization of outliner detectors and interpreters. Extensive use of NRT data ingestion, enrichment, organization and random access storage patterns, make these capabilities possible on top of a Hadoop based ecosystem. The Stratum™ architecture offers the CSO the ability to minimize the time and effects of many cyber threats. Using Big Data technologies for cyber threat analysis is becoming quite common, but the need for outlier detection and interpretation is crucial for enterprise protection.
Recommended
More Related Content
Similar to Near Real-time Outlier Detection and Interpretation - Part 1 by Robert Thorman, AT&T
Similar to Near Real-time Outlier Detection and Interpretation - Part 1 by Robert Thorman, AT&T (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Near Real-time Outlier Detection and Interpretation - Part 1 by Robert Thorman, AT&T
- 1. © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. June 28, 2016 Near Real-time Outlier Detection and Interpretation An Hadoop Based Approach Hadoop Summit 2016 Bob Thorman Principal – Technology Security AT&T Chief Security Organization
- 2. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. 2 Presentation Outline: Brief Context of the Problem of Cyber Threats in our industry Recent History of AT&T Cyber Threat Capabilities Hadoop Based Approach to Threat Analytics Platform Cyber Threat Detection and Interpretation Insider Threat
- 3. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. The Problem of Cyber Threats in Our Industry A Brief Context
- 4. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. 4 Network Scale • ~1M Authenticated users • ~800K user oriented devices • ~1100 security devices on the network (FW, IDS, etc.) • Approximately 5B network events per day – Firewall, Proxy, IDS, SIEM, etc. Facing Alarming Trends Bridging to the Internet • Next Slides The Problem of Cyber Threats in Our Industry
- 5. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. 5 Distributed Reflection DoS (DrDoS) Attack Evolution Attack activity trending up Oct 2013 1900/udp: SSDP 123/udp: NTP 19/udp: chargen 0/udp: packet fragmentation 53/udp: DNS (some legitimate)30 months shown
- 6. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Recent History of AT&T Cyber Threat Protection Capabilities A Need for Big Data
- 7. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. History of AT&T Cyber Threat Protection Capabilities Chief Security Office – 2002 Program concept for millions of records per day – 2005 Program concept tens of millions of records per day – 2016 Big Data concept for tens of billions events/day – 2017 Big Data concepts for trillions events/day Major Big Data Development Milestone – 2008 Beginnings of Accumulo, an implementation of Google™ Bigtable – 2011 Accumulo open sourced to Apache Software Foundation – 2013 AT&T initiates Threat Analytics modernization project – 2014 AT&T initiates deployment of Hadoop-based Threat Analytics Platform Cyber Threat Protection Platform Architecture Evolution – Next slides 7
- 8. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Threat Platform of Yesterday SIEM 8 Source/processing/analytics DBMS/SAN Query
- 9. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Threat Detection and Interpretation Process 9 Architectural Component Ingestion Outlier Detection1 Spark Streaming Detectors1 R Analytics1 Web UI Dashboards Custom Alerting Framework1 Threat Operations 1Area of focus for automation
- 10. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. An Hadoop Based Approach to Threat Analytics Platform Securing AT&T with Hadoop
- 11. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Today’s Platform Details Using An Hadoop Based Platform for Log Management, Threat Analysis, Reporting AT&T approach to use of Hadoop in a Threat Analysis Platform SIEM Raw logs Events, Intelligence, Alarms, Threats Results, Reports, Analytics Source Processing Threat Analytics Platform UI/Visual/Report 11
Editor's Notes
- Introduce Adam Introduce myself
- Work real quick through agenda Just set the stage for an Hadoop based threat analytics platform that has NRT capabilities
- Set the stage for how a typical network in this industry and how much work there is for securing it. Presents an industry problem, not an AT&T problem Address the outside threat to the internal operation of our industry
- Amount of traffic related to reflect based DoS attackers. Illustrates activity on the internet not the attacks against the AT&T perimeter. Hack-ma-geddon Columbia government Spam Hause Syria <- New York Times Target lost 40M credit/debit cards
- Our TAP has evolved a lot over the last few year as we’ve moved into an Hadoop base architecture. I will briefly describe the roadmap. Proprietary technology and lack of extensibility are killers
- Past was SIEM dependent, based on large RDBMS and exclusively dependent on human detection and interpretation. Largely a data reduction system. Industry solution of yesterday.
- The challenge is the cognitive intersection with automation. An environment of innovation. Goal is to automate the security analysis process which are largely cognitive. Granted this is a different use of Hadoop rather than single use data. Its continual ingestion, NRT detections, alerting, etc. Not always a clear problem statement. Spend some time developing the human dependency and cognitive processes Takes a lot of data
- Left to right, we move all the data through various processing platforms into an Hadoop base system for raw log management, data org, management, access, analysis and finally to visualization and reporting.