Video (at YouTube) - http://bit.ly/19TNSTF
Big Data Security Analytics, Data Science and Machine Learning are a few of the new buzzwords that have invaded out industry of late. Most of what we hear are promises of an unicorn-laden, silver-bullet panacea by heavy-handed marketing folks, evoking an expected pushback from the most enlightened members of our community.
This talk will help parse what we as a community need to know and understand about these concepts and help understand where the technical details and actual capabilities of those concepts and also where they fail and how they can be exploited and fooled by an attacker.
The talk will also share results of the author's current ongoing research (on MLSec Project) of applying machine learning techniques to information secuirty monitoring.
BSidesLV 2013 - Using Machine Learning to Support Information SecurityAlex Pinto
Big Data, Data Science, Machine Learning and Analytics are a few of the new buzzwords that have invaded out industry of late. Again we are being sold a unicorn-laden, silver-bullet panacea by heavy handed marketing folks, evoking an expected pushback from the most enlightened members of our community. However, as was the case before, there might just be enough technical meat in there to help out with our security challenges and the overwhelming odds we face everyday. And if so, what do we as a community have to know about these technologies in order to be better professionals? Can we really use the data we have been collecting to help automate our security decision making? Is a robot going to steal my job?
If you are interested in what is behind this marketing buzz and are not scared of a little math, this talk would like to address some insights into applying Machine Learning techniques to data any of us have easy access to, and try to bring home the point that if all of this technology can be used to show us “better” ads in social media and track our behavior online (and a bit more than that) it can also be used to defend our networks as well.
Secure Because Math: A Deep-Dive on Machine Learning-Based Monitoring (#Secur...Alex Pinto
We could all have predicted this with our magical Big Data analytics platforms, but it seems that Machine Learning is the new hotness in Information Security. A great number of startups with ‘cy’ and ‘threat’ in their names that claim that their product will defend or detect more effectively than their neighbour's product "because math". And it should be easy to fool people without a PhD or two that math just works.
Indeed, math is powerful and large scale machine learning is an important cornerstone of much of the systems that we use today. However, not all algorithms and techniques are born equal. Machine Learning is a most powerful tool box, but not every tool can be applied to every problem and that’s where the pitfalls lie.
This presentation will describe the different techniques available for data analysis and machine learning for information security, and discuss their strengths and caveats. The Ghost of Marketing Past will also show how similar the unfulfilled promises of deterministic and exploratory analysis were, and how to avoid making the same mistakes again.
Finally, the presentation will describe the techniques and feature sets that were developed by the presenter on the past year as a part of his ongoing research project on the subject, in particular present some interesting results obtained since the last presentation on DefCon 21, and some ideas that could improve the application of machine learning for use in information security, especially in its use as a helper for security analysts in incident detection and response.
SANS CTI Summit 2016 - Data-Driven Threat Intelligence: SharingAlex Pinto
For the last 18 months, MLSec Project and Niddel collected threat intelligence indicator data from multiple sources in order to make sense of the ecosystem and try to find a measure of efficiency or quality in these feeds. This initiative culminated in the creation of Combine and TIQ-test, two of the open source projects from MLSec Project. These projects have been improved upon for the last year, and are able to gather and compare data from multiple Threat Intelligence sources on the Internet. This research culminated on a talk on SANS CTI Summit 2015 and a contribution to the Verizon DBIR on the same year.
On this talk, we have gathered aggregated usage information from intelligence sharing communities in order to determine if the added interest and "push" towards sharing is really being followed by the companies and if its adoption is putting us in the right track to close these gaps. We propose a new set of metrics on the same vein as TIQ-test to help you understand what does a "healthy" threat intelligence sharing community looks like.
To better illustrate the points and metrics, we will be conducting part of this analysis using usage data from some high-profile threat intelligence platforms and sharing communities, that have been kind enough to contribute with usage data for this research.
Join us in an data-driven analysis of threat intelligence sharing communities and their impact on operational usage of indicators!
Determining the Fit and Impact of CTI Indicators on Your Monitoring Pipeline ...Alex Pinto
Implementing an appropriate data processing pipeline to make good use of your indicators of compromise is a problem that has been successfully addressed over the last few years. Even with all the push of automation and orchestration, a fundamental question remains: WHICH data should I be ingesting in my detection pipelines? There is no lack of data available, shared or not, paid or not. But how to keep my CTI IR team from spinning their wheels on a pile of CTI mud?
This talk will discuss statistical analysis you can do with the CTI indicators you collect and your own network telemetry to define:
- FIT: How appropriate does the CTI data apply to your own traffic. CTI vendors always talk about vertical specific threats, but is that measurable and verifiable?
- IMPACT: How much was your true positive detections assisted by matches and link analysis derived from those CTI feeds
- COVERAGE: Is your current mix of CTI feeds providing "intelligence" on the current threats that you should actually be concerned with?
Those concepts will be introduced and explained with minimal math background needed, and pseudo-code (and real-code!) will be provided to assist organizations to perform those experiments on their own environment. We hope those tools will help attendees to better evaluate the quality of the CTI feeds they ingest from their open sources, paid providers and sharing communities.
Measuring the IQ of your Threat Intelligence Feeds (#tiqtest)Alex Pinto
Follow along with the R Markdown file at http://rpubs.com/alexcpsec/tiq-test-Summer2014-2
Source code available at:
https://github.com/mlsecproject/tiq-test
https://github.com/mlsecproject/tiq-test-Summer2014
https://github.com/mlsecproject/combine
---------
Full Abstract:
Threat Intelligence feeds are now being touted as the saving grace for SIEM and log management deployments, and as a way to supercharge incident detection and even response practices. We have heard similar promises before as an industry, so it is only fair to try to investigate. Since the actual number of breaches and attacks worldwide is unknown, it is impossible to measure how good threat intelligence feeds really are, right? Enter a new scientific breakthrough developed over the last 300 years: statistics!
This presentation will consist of a data-driven analysis of a cross-section of threat intelligence feeds (both open-source and commercial) to measure their statistical bias, overlap, and representability of the unknown population of breaches worldwide. Are they a statistical good measure of the population of "bad stuff" happening out there? Is there even such a thing? How tuned to your specific threat surface are those feeds anyway? Regardless, can we actually make good use of them even if the threats they describe have no overlap with the actual incidents you have been seeing in your environment?
We will provide an open-source tool for attendees to extract, normalize and export data from threat intelligence feeds to use in their internal projects and systems. It will be pre-configured with current OSINT network feed and easily extensible for private or commercial feeds. All the statistical code written and research data used (from the open-source feeds) will be made available in the spirit of reproducible research. The tool itself will be able to be used by attendees to perform the same type of tests on their own data.
Join Alex and Kyle on a journey through the actual real-world usability of threat intelligence to find out which mix of open source and private feeds are right for your organization.
Sharing is Caring: Understanding and Measuring Threat Intelligence Sharing Ef...Alex Pinto
For the last 18 months, MLSec Project and Niddel collaborated to collect threat intelligence indicator data from multiple sources in order to make sense of the ecosystem and try to find a measure of efficiency or quality in these feeds. This initiative culminated in the creation of Combine and TIQ-test, two of the open source projects from MLSec Project. These projects have been improved upon for the last year, and are able to gather and compare data from multiple Threat Intelligence sources on the Internet.
Alex Sieira and his team have gathered aggregated usage information from intelligence sharing communities in order to determine if the added interest and "push" towards sharing is really being followed by the companies and if its adoption is putting us on the right track to close these gaps. He proposes a new set of metrics on the same vein as TIQ-test to help you understand what a "healthy" threat intelligence sharing community looks like.
To better illustrate the points and metrics, Alex will be conducting part of this analysis using usage data from some high-profile threat intelligence platforms and sharing communities that have been kind enough to contribute with usage data for this research.
Biting into the Jawbreaker: Pushing the Boundaries of Threat Hunting AutomationAlex Pinto
Threat Hunting has been commonly definable as a series of investigative actions that should be performed by human teams in order to cover detection gaps where automated tools fail. However, as those techniques become more and more popular and standardized, wouldn't it be the case that we are able to automate a large part of those common threat hunting activities, creating what is basicaly a definition oxymoron?
In this session, we will demonstrate how some IOC-based threat hunting techniques can be automated or constructed to augment human activity by encoding analyst intuition into repeatable data extraction and processing techniques. Those techniques can be used to simplify the triage stage and get actionable information from potential threats with minimal human interaction. The more math-oriented parts will cover descriptive statistics, graph theory, and non-linear scoring techniques on the relationships of known network-based IOCs to an organization's log data.
Our goal here is to demonstrate that by elevating the quality of data available to our automation processes we can effectively simulate "analyst intuition" on some of the more time consuming aspects of network threat hunting. IR teams can then theoretically more productive as soon as the initial triage stages, with data products that provide a “sixth sense” on what events are the ones worth of additional analyst time.
Data-Driven Threat Intelligence: Metrics on Indicator Dissemination and SharingAlex Pinto
For the past 18 months, Niddel has been collecting threat intelligence indicator data from multiple sources in order to make sense of the ecosystem and try to find a measure of efficiency or quality in these feeds. This initiative culminated in the creation of Combine and TIQ-test, two of the open source projects from MLSec Project. These projects have been improved upon for the last year and are able to gather and compare data from multiple Threat Intelligence sources on the Internet.
We take this analysis a step further and extract insights form more than 12 months of collected threat intel data to verify the overlap and uniqueness of those sources. If we are able to find enough overlap, there could be a strategy that could put together to acquire an optimal number of feeds, but as Niddel demonstrated on the 2015 Verizon DBIR, that is not the case.
We also gathered aggregated usage information from intelligence sharing communities in order to determine if the added interest and "push" towards sharing is really being followed by the companies and if its adoption is putting us in the right track to close these gaps.
Join us in an data-driven analysis of over an year of collected Threat Intelligence indicators and their sharing communities!
BSidesLV 2013 - Using Machine Learning to Support Information SecurityAlex Pinto
Big Data, Data Science, Machine Learning and Analytics are a few of the new buzzwords that have invaded out industry of late. Again we are being sold a unicorn-laden, silver-bullet panacea by heavy handed marketing folks, evoking an expected pushback from the most enlightened members of our community. However, as was the case before, there might just be enough technical meat in there to help out with our security challenges and the overwhelming odds we face everyday. And if so, what do we as a community have to know about these technologies in order to be better professionals? Can we really use the data we have been collecting to help automate our security decision making? Is a robot going to steal my job?
If you are interested in what is behind this marketing buzz and are not scared of a little math, this talk would like to address some insights into applying Machine Learning techniques to data any of us have easy access to, and try to bring home the point that if all of this technology can be used to show us “better” ads in social media and track our behavior online (and a bit more than that) it can also be used to defend our networks as well.
Secure Because Math: A Deep-Dive on Machine Learning-Based Monitoring (#Secur...Alex Pinto
We could all have predicted this with our magical Big Data analytics platforms, but it seems that Machine Learning is the new hotness in Information Security. A great number of startups with ‘cy’ and ‘threat’ in their names that claim that their product will defend or detect more effectively than their neighbour's product "because math". And it should be easy to fool people without a PhD or two that math just works.
Indeed, math is powerful and large scale machine learning is an important cornerstone of much of the systems that we use today. However, not all algorithms and techniques are born equal. Machine Learning is a most powerful tool box, but not every tool can be applied to every problem and that’s where the pitfalls lie.
This presentation will describe the different techniques available for data analysis and machine learning for information security, and discuss their strengths and caveats. The Ghost of Marketing Past will also show how similar the unfulfilled promises of deterministic and exploratory analysis were, and how to avoid making the same mistakes again.
Finally, the presentation will describe the techniques and feature sets that were developed by the presenter on the past year as a part of his ongoing research project on the subject, in particular present some interesting results obtained since the last presentation on DefCon 21, and some ideas that could improve the application of machine learning for use in information security, especially in its use as a helper for security analysts in incident detection and response.
SANS CTI Summit 2016 - Data-Driven Threat Intelligence: SharingAlex Pinto
For the last 18 months, MLSec Project and Niddel collected threat intelligence indicator data from multiple sources in order to make sense of the ecosystem and try to find a measure of efficiency or quality in these feeds. This initiative culminated in the creation of Combine and TIQ-test, two of the open source projects from MLSec Project. These projects have been improved upon for the last year, and are able to gather and compare data from multiple Threat Intelligence sources on the Internet. This research culminated on a talk on SANS CTI Summit 2015 and a contribution to the Verizon DBIR on the same year.
On this talk, we have gathered aggregated usage information from intelligence sharing communities in order to determine if the added interest and "push" towards sharing is really being followed by the companies and if its adoption is putting us in the right track to close these gaps. We propose a new set of metrics on the same vein as TIQ-test to help you understand what does a "healthy" threat intelligence sharing community looks like.
To better illustrate the points and metrics, we will be conducting part of this analysis using usage data from some high-profile threat intelligence platforms and sharing communities, that have been kind enough to contribute with usage data for this research.
Join us in an data-driven analysis of threat intelligence sharing communities and their impact on operational usage of indicators!
Determining the Fit and Impact of CTI Indicators on Your Monitoring Pipeline ...Alex Pinto
Implementing an appropriate data processing pipeline to make good use of your indicators of compromise is a problem that has been successfully addressed over the last few years. Even with all the push of automation and orchestration, a fundamental question remains: WHICH data should I be ingesting in my detection pipelines? There is no lack of data available, shared or not, paid or not. But how to keep my CTI IR team from spinning their wheels on a pile of CTI mud?
This talk will discuss statistical analysis you can do with the CTI indicators you collect and your own network telemetry to define:
- FIT: How appropriate does the CTI data apply to your own traffic. CTI vendors always talk about vertical specific threats, but is that measurable and verifiable?
- IMPACT: How much was your true positive detections assisted by matches and link analysis derived from those CTI feeds
- COVERAGE: Is your current mix of CTI feeds providing "intelligence" on the current threats that you should actually be concerned with?
Those concepts will be introduced and explained with minimal math background needed, and pseudo-code (and real-code!) will be provided to assist organizations to perform those experiments on their own environment. We hope those tools will help attendees to better evaluate the quality of the CTI feeds they ingest from their open sources, paid providers and sharing communities.
Measuring the IQ of your Threat Intelligence Feeds (#tiqtest)Alex Pinto
Follow along with the R Markdown file at http://rpubs.com/alexcpsec/tiq-test-Summer2014-2
Source code available at:
https://github.com/mlsecproject/tiq-test
https://github.com/mlsecproject/tiq-test-Summer2014
https://github.com/mlsecproject/combine
---------
Full Abstract:
Threat Intelligence feeds are now being touted as the saving grace for SIEM and log management deployments, and as a way to supercharge incident detection and even response practices. We have heard similar promises before as an industry, so it is only fair to try to investigate. Since the actual number of breaches and attacks worldwide is unknown, it is impossible to measure how good threat intelligence feeds really are, right? Enter a new scientific breakthrough developed over the last 300 years: statistics!
This presentation will consist of a data-driven analysis of a cross-section of threat intelligence feeds (both open-source and commercial) to measure their statistical bias, overlap, and representability of the unknown population of breaches worldwide. Are they a statistical good measure of the population of "bad stuff" happening out there? Is there even such a thing? How tuned to your specific threat surface are those feeds anyway? Regardless, can we actually make good use of them even if the threats they describe have no overlap with the actual incidents you have been seeing in your environment?
We will provide an open-source tool for attendees to extract, normalize and export data from threat intelligence feeds to use in their internal projects and systems. It will be pre-configured with current OSINT network feed and easily extensible for private or commercial feeds. All the statistical code written and research data used (from the open-source feeds) will be made available in the spirit of reproducible research. The tool itself will be able to be used by attendees to perform the same type of tests on their own data.
Join Alex and Kyle on a journey through the actual real-world usability of threat intelligence to find out which mix of open source and private feeds are right for your organization.
Sharing is Caring: Understanding and Measuring Threat Intelligence Sharing Ef...Alex Pinto
For the last 18 months, MLSec Project and Niddel collaborated to collect threat intelligence indicator data from multiple sources in order to make sense of the ecosystem and try to find a measure of efficiency or quality in these feeds. This initiative culminated in the creation of Combine and TIQ-test, two of the open source projects from MLSec Project. These projects have been improved upon for the last year, and are able to gather and compare data from multiple Threat Intelligence sources on the Internet.
Alex Sieira and his team have gathered aggregated usage information from intelligence sharing communities in order to determine if the added interest and "push" towards sharing is really being followed by the companies and if its adoption is putting us on the right track to close these gaps. He proposes a new set of metrics on the same vein as TIQ-test to help you understand what a "healthy" threat intelligence sharing community looks like.
To better illustrate the points and metrics, Alex will be conducting part of this analysis using usage data from some high-profile threat intelligence platforms and sharing communities that have been kind enough to contribute with usage data for this research.
Biting into the Jawbreaker: Pushing the Boundaries of Threat Hunting AutomationAlex Pinto
Threat Hunting has been commonly definable as a series of investigative actions that should be performed by human teams in order to cover detection gaps where automated tools fail. However, as those techniques become more and more popular and standardized, wouldn't it be the case that we are able to automate a large part of those common threat hunting activities, creating what is basicaly a definition oxymoron?
In this session, we will demonstrate how some IOC-based threat hunting techniques can be automated or constructed to augment human activity by encoding analyst intuition into repeatable data extraction and processing techniques. Those techniques can be used to simplify the triage stage and get actionable information from potential threats with minimal human interaction. The more math-oriented parts will cover descriptive statistics, graph theory, and non-linear scoring techniques on the relationships of known network-based IOCs to an organization's log data.
Our goal here is to demonstrate that by elevating the quality of data available to our automation processes we can effectively simulate "analyst intuition" on some of the more time consuming aspects of network threat hunting. IR teams can then theoretically more productive as soon as the initial triage stages, with data products that provide a “sixth sense” on what events are the ones worth of additional analyst time.
Data-Driven Threat Intelligence: Metrics on Indicator Dissemination and SharingAlex Pinto
For the past 18 months, Niddel has been collecting threat intelligence indicator data from multiple sources in order to make sense of the ecosystem and try to find a measure of efficiency or quality in these feeds. This initiative culminated in the creation of Combine and TIQ-test, two of the open source projects from MLSec Project. These projects have been improved upon for the last year and are able to gather and compare data from multiple Threat Intelligence sources on the Internet.
We take this analysis a step further and extract insights form more than 12 months of collected threat intel data to verify the overlap and uniqueness of those sources. If we are able to find enough overlap, there could be a strategy that could put together to acquire an optimal number of feeds, but as Niddel demonstrated on the 2015 Verizon DBIR, that is not the case.
We also gathered aggregated usage information from intelligence sharing communities in order to determine if the added interest and "push" towards sharing is really being followed by the companies and if its adoption is putting us in the right track to close these gaps.
Join us in an data-driven analysis of over an year of collected Threat Intelligence indicators and their sharing communities!
From Threat Intelligence to Defense Cleverness: A Data Science Approach (#tid...Alex Pinto
This session will center on a market-centric and technological exploration of commercial and open-source threat intelligence feeds that are becoming common to be offered as a way to improve the defense capabilities of organizations.
While not all Threat Intelligence can be represented as "indicator feeds", this space has enough market attention that it deserves a proper scientific, evidence-based investigation so that practitioners and decision makers can maximize the results they are able to get for the data they have available.
The presentation will consist of a data-driven analysis of a cross-section of threat intelligence feeds (both open-source and commercial) to measure their statistical bias, overlap, and representability of the unknown population of breaches worldwide. All the statistical code written and research data used (from the publicly available feeds) will be made available in the spirit of reproducible research. The tool itself will be able to be used by attendees to perform the same type of tests on their own data (called tiq-test).
Some of the important questions and answers that emerge in this presentation include:
"Are Threat Intelligence Feeds a statistical good measure of the population of 'bad stuff' happening out there? Is there even such a thing?"
"How tuned to YOUR specific threat surface are those feeds?"
"Can we actually make good use of them even if the threats they describe have no overlap with the actual incidents you have been seeing in your environment? (hint: probably not)"
We will provide an open-source tool for attendees to extract, normalize and export data from threat intelligence feeds to use in their internal projects and systems. It will be pre-configured with current publicly available network feeds and easily extensible for private or commercial feeds (called combine).
Towards a Threat Hunting Automation Maturity ModelAlex Pinto
Threat Hunting has been commonly definable as a series of investigative actions that should be performed by analyst teams to cover detection gaps where automated tools fail. However, as those techniques become more and more widespread and standardized, wouldn’t it be the case that we can automate a large part of those threat hunting activities, creating a definition oxymoron?
In this session, we will demonstrate how some threat hunting techniques can be automated or constructed to augment human activity by encoding analyst intuition into repeatable data extraction and processing technologies. Those techniques can be used to simplify the triage stage and get actionable information from potential threats with minimal human interaction. We then present a Hunting Automation Maturity Model (HAMM) that organizes these techniques around capability milestones, including internal and external context and analytical tooling.
Beyond Matching: Applying Data Science Techniques to IOC-based DetectionAlex Pinto
There is no doubt that indicators of compromise (IOCs) are here to stay. However, even the most mature incident response (IR) teams are currently mainly focused on matching known indicators to their captured traffic or logs. The real “eureka” moments of using threat intelligence mostly come out of analyst intuition. You know, the ones that are almost impossible to hire.
In this session, we show you how you can apply descriptive statistics, graph theory, and non-linear scoring techniques on the relationships of known network IOCs to log data. Learn how to use those techniques to empower IR teams to encode analyst intuition into repeatable data techniques that can be used to simplify the triage stage and get actionable information with minimal human interaction.
With these results, we can make IR teams more productive as soon as the initial triage stages, by providing them data products that provide a “sixth sense” on what events are the ones worth analyst time. They also make painfully evident which IOC feeds an organization consume that are being helpful to their detection process and which ones are not.
Data-Driven Threat Intelligence: Useful Methods and Measurements for Handling...Alex Pinto
This session will consist of a technological exploration of commercial and open-source threat intelligence feeds that are commonly offered as a way to improve the capabilities of incident response teams. While not all Threat Intelligence can be represented as "indicator feeds", this space has enough market attention that it deserves a proper scientific, evidence-based investigation so that practitioners and decision makers can maximize the results they are able to get for the data they have available.
We will present a data-driven analysis of a cross-section of threat intelligence feeds (both open-source and commercial) to measure their statistical bias, overlap, and representability of the unknown population of breaches worldwide, in addition to some tidbits as indicator age and uniqueness across feeds. All the statistical code written and research data used (from the publicly available feeds) will be made available in the spirit of reproducible research. The tool itself (tiq-test) will be able to be used by attendees to perform the same type of tests on their own data.
We will also provide an additional open-source tool (combine) for attendees to extract, normalize and export data from threat intelligence feeds to use in their internal projects and systems. It will be pre-configured with a good mix of current publicly available network feeds and easily extensible for private or commercial feeds.
Data protection is at the center of a mature organizational information security strategy. Encryption plays an important role in that strategy to effectively protect data, even after other lines of defense have been compromised.
Unfortunately, there are many factors complicating the when, where and how of successfully using encryption technologies:
IT Operation Analytic for security- MiSSconf(sp1)stelligence
IT Operation Analytic: Using Anomaly Detection , Unsupervised Machine Learning, to distinct normal and abnormal behavior and enhance efficiency of SIEM detection and alert capability.
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start ‘hunting’ for signs of compromises and anomalies in our own environments.
In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company.
Visualization. Data science. No machine learning. But pretty pictures.What is internal threat intelligence? Check out http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225
Delivering Security Insights with Data Analytics and VisualizationRaffael Marty
It's an interesting exercise to look back to the year 2000 to see how we approached cyber security. We just started to realize that data might be a useful currency, but for the most part, security pursued preventative avenues, such as firewalls, intrusion prevention systems, and anti-virus. With the advent of log management and security incident and event management (SIEM) solutions we started to gather gigabytes of sensor data and correlate data from different sensors to improve on their weaknesses and accelerate their strengths. But fundamentally, such solutions didn't scale that well and struggled to deliver real security insight.
Today, cybersecurity wouldn't work anymore without large scale data analytics and machine learning approaches, especially in the realm of malware classification and threat intelligence. Nonetheless, we are still just scratching the surface and learning where the real challenges are in data analytics for security.
This talk will go on a journey of big data in cybersecurity, exploring where big data has been and where it must go to make a true difference. We will look at the potential of data mining, machine learning, and artificial intelligence, as well as the boundaries of these approaches. We will also look at both the shortcomings and potential of data visualization and the human computer interface. It is critical that today's systems take into account the human expert and, most importantly, provide the right data.
From Threat Intelligence to Defense Cleverness: A Data Science Approach (#tid...Alex Pinto
This session will center on a market-centric and technological exploration of commercial and open-source threat intelligence feeds that are becoming common to be offered as a way to improve the defense capabilities of organizations.
While not all Threat Intelligence can be represented as "indicator feeds", this space has enough market attention that it deserves a proper scientific, evidence-based investigation so that practitioners and decision makers can maximize the results they are able to get for the data they have available.
The presentation will consist of a data-driven analysis of a cross-section of threat intelligence feeds (both open-source and commercial) to measure their statistical bias, overlap, and representability of the unknown population of breaches worldwide. All the statistical code written and research data used (from the publicly available feeds) will be made available in the spirit of reproducible research. The tool itself will be able to be used by attendees to perform the same type of tests on their own data (called tiq-test).
Some of the important questions and answers that emerge in this presentation include:
"Are Threat Intelligence Feeds a statistical good measure of the population of 'bad stuff' happening out there? Is there even such a thing?"
"How tuned to YOUR specific threat surface are those feeds?"
"Can we actually make good use of them even if the threats they describe have no overlap with the actual incidents you have been seeing in your environment? (hint: probably not)"
We will provide an open-source tool for attendees to extract, normalize and export data from threat intelligence feeds to use in their internal projects and systems. It will be pre-configured with current publicly available network feeds and easily extensible for private or commercial feeds (called combine).
Towards a Threat Hunting Automation Maturity ModelAlex Pinto
Threat Hunting has been commonly definable as a series of investigative actions that should be performed by analyst teams to cover detection gaps where automated tools fail. However, as those techniques become more and more widespread and standardized, wouldn’t it be the case that we can automate a large part of those threat hunting activities, creating a definition oxymoron?
In this session, we will demonstrate how some threat hunting techniques can be automated or constructed to augment human activity by encoding analyst intuition into repeatable data extraction and processing technologies. Those techniques can be used to simplify the triage stage and get actionable information from potential threats with minimal human interaction. We then present a Hunting Automation Maturity Model (HAMM) that organizes these techniques around capability milestones, including internal and external context and analytical tooling.
Beyond Matching: Applying Data Science Techniques to IOC-based DetectionAlex Pinto
There is no doubt that indicators of compromise (IOCs) are here to stay. However, even the most mature incident response (IR) teams are currently mainly focused on matching known indicators to their captured traffic or logs. The real “eureka” moments of using threat intelligence mostly come out of analyst intuition. You know, the ones that are almost impossible to hire.
In this session, we show you how you can apply descriptive statistics, graph theory, and non-linear scoring techniques on the relationships of known network IOCs to log data. Learn how to use those techniques to empower IR teams to encode analyst intuition into repeatable data techniques that can be used to simplify the triage stage and get actionable information with minimal human interaction.
With these results, we can make IR teams more productive as soon as the initial triage stages, by providing them data products that provide a “sixth sense” on what events are the ones worth analyst time. They also make painfully evident which IOC feeds an organization consume that are being helpful to their detection process and which ones are not.
Data-Driven Threat Intelligence: Useful Methods and Measurements for Handling...Alex Pinto
This session will consist of a technological exploration of commercial and open-source threat intelligence feeds that are commonly offered as a way to improve the capabilities of incident response teams. While not all Threat Intelligence can be represented as "indicator feeds", this space has enough market attention that it deserves a proper scientific, evidence-based investigation so that practitioners and decision makers can maximize the results they are able to get for the data they have available.
We will present a data-driven analysis of a cross-section of threat intelligence feeds (both open-source and commercial) to measure their statistical bias, overlap, and representability of the unknown population of breaches worldwide, in addition to some tidbits as indicator age and uniqueness across feeds. All the statistical code written and research data used (from the publicly available feeds) will be made available in the spirit of reproducible research. The tool itself (tiq-test) will be able to be used by attendees to perform the same type of tests on their own data.
We will also provide an additional open-source tool (combine) for attendees to extract, normalize and export data from threat intelligence feeds to use in their internal projects and systems. It will be pre-configured with a good mix of current publicly available network feeds and easily extensible for private or commercial feeds.
Data protection is at the center of a mature organizational information security strategy. Encryption plays an important role in that strategy to effectively protect data, even after other lines of defense have been compromised.
Unfortunately, there are many factors complicating the when, where and how of successfully using encryption technologies:
IT Operation Analytic for security- MiSSconf(sp1)stelligence
IT Operation Analytic: Using Anomaly Detection , Unsupervised Machine Learning, to distinct normal and abnormal behavior and enhance efficiency of SIEM detection and alert capability.
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start ‘hunting’ for signs of compromises and anomalies in our own environments.
In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company.
Visualization. Data science. No machine learning. But pretty pictures.What is internal threat intelligence? Check out http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225
Delivering Security Insights with Data Analytics and VisualizationRaffael Marty
It's an interesting exercise to look back to the year 2000 to see how we approached cyber security. We just started to realize that data might be a useful currency, but for the most part, security pursued preventative avenues, such as firewalls, intrusion prevention systems, and anti-virus. With the advent of log management and security incident and event management (SIEM) solutions we started to gather gigabytes of sensor data and correlate data from different sensors to improve on their weaknesses and accelerate their strengths. But fundamentally, such solutions didn't scale that well and struggled to deliver real security insight.
Today, cybersecurity wouldn't work anymore without large scale data analytics and machine learning approaches, especially in the realm of malware classification and threat intelligence. Nonetheless, we are still just scratching the surface and learning where the real challenges are in data analytics for security.
This talk will go on a journey of big data in cybersecurity, exploring where big data has been and where it must go to make a true difference. We will look at the potential of data mining, machine learning, and artificial intelligence, as well as the boundaries of these approaches. We will also look at both the shortcomings and potential of data visualization and the human computer interface. It is critical that today's systems take into account the human expert and, most importantly, provide the right data.
Methods and techniques for monitoring, surveillance and profiling of cyberspace activities are available to state actors and criminal operators as well as to skilled individuals.
Civilian individuals without extensive knowledge and exposure to counter-intelligence techniques and Operations Security (OPSEC) protocols are vulnerable and helpless against these privacy breaches. This represents significant gap between actors and their victims – gap that can be bridged only through education.
The primary objective of this workshop oriented activity is to educate people operating in above-average risk situations in cyberspace and to bridge the unfair gap between them and malicious actors. Secondary objective aims at understanding how cyber security can be taught to non-security minded people in efficient way.
Managing Next Generation Threats to Cyber SecurityPriyanka Aash
The emergence of next generation technology into the cyber security space has added complications and challenges on several levels. When we talk about next generation technologies we should mean those associated directly with artificial intelligence (AI) and associated components such as machine learning (ML). Unfortunately, many organizations opt to hype current generation products as next gen. In this workshop we will begin by exploring what we need to know about AI and its components. We will dispense with the marketing hype and get down to the facts. Then we will look in detail at a few available tools that truly are next gen - and what makes them next gen - followed by a discussion of where the adversary is going with AI, ML and other next gen technologies. We will wrap up with research from my upcoming book which discusses the collision between the law and cyber science. In this section we also will address some governance issues that you need to know
Presented by Paul Wilson, Director General of APNIC and Chair of APrIGF Multistakeholder Steering Group at the Asia Pacific Internet Leadership Program as part of 2016 APrIGF Taipei
UMS Cybersecurity Awareness Seminar: Cybersecurity - Lessons learned from sec...APNIC
APNIC Senior Security Specialist Adli Wahid provides some useful findings of lessons learned from security incidents at the UMS Cybersecurity Awareness Seminar, held online on 25 October 2021.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Search and Society: Reimagining Information Access for Radical Futures
Applying Machine Learning to Network Security Monitoring - BayThreat 2013
1. Applying
Machine
Learning
to
Network
Security
Monitoring
Alexandre
Pinto
Chief
Data
Scien4st
|
MLSec
Project
@alexcpsec
@MLSecProject!
2. WARNING!
• This
is
a
talk
about
BUILDING
not
breaking
– NO
systems
were
harmed
on
the
development
of
this
talk.
– This
is
NOT
about
1337
Android
Malware
• Only
thing
we
are
likely
to
break
here
is
the
4me
limit
on
the
talk
• This
talk
includes
more
MATH
than
the
daily
recommended
intake
by
the
FDA.
• All
stunts
described
in
this
talk
were
performed
by
trained
professionals.!
3. Who's
Alex?
• 13
years
in
Informa4on
Security,
done
a
liRle
bit
of
everything.
• Past
7
or
so
years
leading
security
consultancy
and
monitoring
teams
in
Brazil,
London
and
the
US.
– If
there
is
any
way
a
SIEM
can
hurt
you,
it
did
to
me.
• Researching
machine
learning
and
data
science
in
general
for
the
past
year
or
so
and
presen4ng
about
the
intersec4on
of
it
and
Infosec
throughout
the
year.
• Created
MLSec
Project
in
July
2013
to
give
structure
to
the
research
being
done.
4. Agenda
• Defini4ons
• Big
Data
• Data
Science
• Machine
Learning
•
•
•
•
•
Y
U
DO
DIS?
Network
Security
Monitoring
PoC
||
GTFO
Feature
Intui4on
How
to
get
started?
8. (Security)
Data
ScienEst
• “Data
Scien4st
(n.):
Person
who
is
beRer
at
sta4s4cs
than
any
so`ware
engineer
and
beRer
at
so`ware
engineering
than
any
sta4s4cian.”
-‐-‐
Josh
Willis,
Cloudera
Data
Science
Venn
Diagram
by
Drew
Conway!
9. Enter
Machine
Learning
• “Machine
learning
systems
automa4cally
learn
programs
from
data”
(*)
• You
don’t
really
code
the
program,
but
it
is
inferred
from
data.
• Intui4on
of
trying
to
mimic
the
way
the
brain
learns:
that's
where
terms
like
ar#ficial
intelligence
come
from.
!
(*)
CACM
55(10)
-‐
A
Few
Useful
Things
to
Know
about
Machine
Learning
(Domingos
2012)
13. ConsideraEons
on
Data
Gathering
• Models
will
(generally)
get
beRer
with
more
data
– But
we
always
have
to
consider
bias
and
variance
as
we
select
our
data
points
– Also
adversaries
–
we
may
be
force
fed
“bad
data”,
find
signal
in
weird
noise
or
design
bad
(or
exploitable)
features
• “I’ve
got
99
problems,
but
data
ain’t
one”!
Domingos,
2012
Abu-‐Mostafa,
Caltech,
2012
15. Y
U
DO
DIS?
• Common
reac4ons
from
Security
Professionals:
• “Eh,
cool…”
*blank
stare*
*walks
away*
• “Are
you
high,
bro?”
• “Why
aren’t
you
doing
some
cool
research
like
Android
Malware?”
17. Security
ApplicaEons
of
ML
• Fraud
detec4on
systems:
– Is
what
he
just
did
consistent
with
past
behavior?
• Network
anomaly
detec4on
(?):
– More
like
bad
sta4s4cal
analysis
– Did
not
advance
a
lot,
IMO
• Predic4ng
likelihood
of
aRack
actors
– Create
different
predic4ve
models
and
chain
them
to
gain
more
confidence
in
each
step.!
• SPAM
filters
18. ConsideraEons
on
Data
Gathering
• Adversaries
-‐
Exploi4ng
the
learning
process
• Understand
the
model,
understand
the
machine,
and
you
can
circumvent
it
• Something
InfoSec
community
knows
very
well
• Any
predic4ve
model
on
InfoSec
will
be
pushed
to
the
limit
• Again,
think
back
on
the
way
SPAM
engines
evolved.!
20. CorrelaEon
Rules:
A
Primer
• Rules
in
a
SIEM
solu4on
invariably
are:
– “Something”
has
happened
“x”
4mes;
– “Something”
has
happened
and
other
“something2”
has
happened,
with
some
rela4onship
(4me,
same
fields,
etc)
between
them.
• Configuring
SIEM
=
iterate
on
combina4ons
un4l:
– Customer
or
management
is
foole..
I
mean
sa4sfied;
– Consul4ng
money
runs
out
• Behavioral
rules
(anomaly
detec4on)
helps
a
bit
with
the
“x”s,
but
s4ll,
very
laborious
and
4me
consuming.!
21. Kinds
of
Network
Security
Monitoring
• Alert-‐based:
– “Tradi4onal”
log
management
– SIEM
– Using
“Threat
Intelligence”
(i.e
blacklists)
for
about
a
year
or
so
– Lack
of
context
– Low
effec4veness
– You
get
the
results
handed
over
to
you
• Explora4on-‐based:
– Network
Forensics
tools
(2/3
years
ago)
– Elas4c
Search
based
LM
systems
– High
effec4veness
– Lots
of
people
necessary
– Lots
of
HIGHLY
trained
people
• Big
Data
Security
Analy4cs
(BDSA):
– Run
explora4on-‐based
monitoring
on
Hadoop
– More
like
Big
Data
Security
Monitoring
(BDSM)
25. PoC
||
GTFO
• We
developed
a
set
of
algorithms
to
detect
malicious
behavior
from
log
entries
of
firewall
blocks
• Over
6
months
of
data
from
SANS
DShield
(thanks,
guys!)
• A`er
a
lot
of
sta4s4cal-‐based
math
(true
posi4ve
ra4o,
true
nega4ve
ra4o,
odds
likelihood),
it
could
pinpoint
actors
that
would
be
13x-‐18x
more
likely
to
aRack
you.
• Today
more
like
30x
on
the
SANS
data,
and
finding
around
80%
of
“badness”
in
par4cipant
deployments.!
26. Feature
IntuiEon:
IP
Proximity
• Assump4ons
to
aggregate
the
data
• Correla4on
/
proximity
/
similarity
BY
BEHAVIOR
• “Bad
Neighborhoods”
concept:
– Spamhaus
x
CyberBunker
– Google
Report
(June
2013)
– Moura
2013
• Group
by
Geoloca4on
• Group
by
Netblock
(/16,
/24)
• Group
by
ASN
– (thanks,
Team
Cymru)!
27. 0
10
MULTICAST
AND
FRIENDS
You
are
here!
CN,
BR,
TH
Map
of
the
Internet
• (Hilbert
Curve)
• Block
port
22
• 2013-‐07-‐20
CN
127
RU
28. Feature
IntuiEon:
Temporal
Decay
• Even
bad
neighborhoods
renovate:
– ARackers
may
change
ISPs/proxies
– Botnets
may
be
shut
down
/
relocate
– A
liRle
paranoia
is
Ok,
but
not
EVERYONE
is
out
to
get
you
(at
least
not
all
at
once)!
• As
days
pass,
let's
forget,
bit
by
bit,
who
aRacked
• Last
4me
I
saw
this
actor,
and
how
o`en
did
I
see
them!
29. MLSec
Project
• Behavior:
block
on
port
22
• Trial
inference
on
100k
IP
addresses
per
Class
A
subnet
• Logarithm
scale:
brightest
4les
are
10
to
1000
4mes
more
likely
to
aRack.
30. Feature
IntuiEon:
DNS
features
• Who
resolves
to
this
IP
address?
• Number
of
domains
that
resolve
to
the
IP
address
• Distribu4on
of
their
life4me
• Entropy,
size,
ccTLDs
• Registrar
informa4on
• Reverse
DNS
informa4on…
• History
of
DNS
registra4on…
• (Thanks,
DNSDB!)
31. Training
the
Model
• YAY!
We
have
a
bunch
of
numbers
per
IP
address/domain!
• How
do
you
define
what
is
malicious
or
not?
• “Advanced
exper4se
in
both
informa4on
security
and
data
science
will
be
a
necessary
ingredient
in
enabling
accurate
discrimina4on
between
malicious
and
benign
ac4vity.
“
-‐
Anton
Chuvakin,
Gartner
• Kinda
easy
for
security
tools
(if
you
trust
them)
• Web
applica4on
logs
need
deeper
sta4s4cal
analysis
• Not
normal
/
standard
devia4on
thing
!
32. How
do
I
get
started
on
this?
• Programming
is
a
must
(Python
/
R)
• Sta4s4cal
knowledge
keeps
you
from
making
dumb
mistakes
• Specific
machine
learning
courses
and
books:
– Coursera
(ML/
Data
Analysis
/
Data
Science)
• Prac4ce,
Prac4ce,
Prac4ce:
– Explore
your
data!
–
(Security
Onion)
– Kaggle
– KDD,
VAST,
VizSec!
33. MLSec
Project
• Sign
up,
send
logs,
receive
reports
generated
by
machine
learning
models!
• Working
with
several
companies
on
trying
out
these
models
on
their
environment
with
their
data
• We
are
hiring
(KINDA)
• Visit
h]ps://www.mlsecproject.org
,
message
@MLSecProject
or
just
e-‐mail
me.!
34. MLSec
Project
-‐
Current
Research
• Inbound
aRacks
on
exposed
services
(DEFCON/BH
2013):
– Informa4on
from
inbound
connec4ons
on
firewalls,
IPS,
WAFs
– Feature
extrac4on
and
supervised
learning
• Malware
Distribu4on
and
Botnets:
– Informa4on
from
outbound
connec4ons
on
firewalls,
DNS
and
Web
Proxy
– Ini4al
labeling
provided
by
intelligence
feeds
and
AV/an4-‐malware
– Semi-‐supervised
learning
involved
• Kill-‐chain
Ensemble
Models:
– Increased
precision
by
composing
different
behaviors
– Web
server
path
-‐>
go
through
Firewall,
then
IPS,
then
WAF
– Early
confirma4on
of
aRack
failure
or
success
35. Thanks!
• Q&A?
• Feedback?
Alexandre
Pinto
@alexcpsec
@MLSecProject
hRps://www.mlsecproject.org/
"
Essen4ally,
all
models
are
wrong,
but
some
are
useful."
-‐
George
E.
P.
Box