PDD 2019 Career development informed by social justice - Ann Delauzun
ASCO TAP Social Science Research
1. UNCLASSIFIED
UNCLASSIFIED
Ms. Jennifer Perry Representing
The ASCO Threat
Anticipation Team
February 8, 2011
Approved for public release; distribution is unlimited.
ASCO’s Social Sciences Research
Agenda: A Look Back
2. UNCLASSIFIED
UNCLASSIFIED
2
Outline of Discussion
• Who We Are and What We Do.
• Why a Look Back?
• A Look Back.
• 2002-2006: How We Started.
• 2006-2010: What We’ve Done Since.
• 2010-2011: Our Current Focus.
• Lessons Learned.
3. UNCLASSIFIED
UNCLASSIFIED
3
Who We Are
The Advanced Systems and Concepts Office
(ASCO) emphasizes the identification,
integration, and further development of
leading strategic thinking and analysis on the
most intractable problems related to
combating weapons of mass destruction.
ASCO ESSENTIALLY ACTS AS DTRA’S INTERNAL STRATEGIC
STUDIES OFFICE. IT SERVES DTRA, BUT ALSO THE BROADER
NATIONAL SECURITY COMMUNITY.
4. UNCLASSIFIED
UNCLASSIFIED
4
ASCO’s Social Sciences Research
Program Objective: What We Do
Identify and develop social sciences-based
research, and analyses to support the
anticipation and reduction of weapons of
mass destruction (WMD) and related threats
along a rolling long-term horizon.
INCLUDES ASSESSMENTS OF THE CHALLENGES OF LEVERAGING
COMPUTATIONAL SOCIAL MODELING APPROACHES TO ASSESS
AND ANTICIPATE NATIONAL SECURITY THREATS.
5. UNCLASSIFIED
UNCLASSIFIED
5
General Caveats About the “Research
Program”
• This research is but one small focus area for
the office.
• Ideas for studies are generally generated “in
house.”
• Program is not requirement-driven.
6. UNCLASSIFIED
UNCLASSIFIED
6
Why a Look Back?
• Niche focus can offer broader insights and lessons
learned.
• 2011: End of the current program, but research activities can
be used as a basis for initiatives others undertake.
• A review of activities from 2002 to 2011 can offer:
• Ideas on productive ways to do this research.
• Lessons learned about research challenges.
• An understanding of research gaps and “starting points” for
future research.
7. UNCLASSIFIED
UNCLASSIFIED
7
A Look Back: How We Started
• 2002: Research challenge from DTRA Director.
• Know the research base and help move it forward.
• Assess the existing research and current and emerging
advances in modeling to anticipate national security threats.
• Do modeling research and development to demonstrate how
models can be used to aid decision makers.
• 2002-2006: Response to the challenge.
• Prototype modeling (labs, commercial contractors).
• Neural net, expected utility model, multi-agent simulation
model, cultural simulation model.
• Joint Threat Anticipation Center.
8. UNCLASSIFIED
UNCLASSIFIED
8
A Look Back: 2006-2010
• 2006: Program restructuring.
• Get out of the model development business.
• Focus more on grand challenges in doing modeling
and using models and their findings.
• Explore the value of social science research to assess,
anticipate, and reduce WMD threats, specifically.
• 2006-2010: Program focus areas
KNOW THE RESEARCH BASE ASSESS GRAND
CHALLENGES IN “USING” IT APPLY IT TO ASSESS,
ANTICIPATE, AND REDUCE WMD THREATS.
9. UNCLASSIFIED
UNCLASSIFIED
9
A Look Back at 2006-2010:
General Themes
• WMD/E terrorism:
• Research on the motivations, intentions and need to also
consider capabilities and feasibilities.
• Challenges and relevancy of academic research.
• Assess the limits and potentials of research and
advances in “hot areas” of interest for USG.
• Strategic culture and “tailored” analysis.
• Social network theory.
• Computational social modeling.
• Qualitative and quantitative prediction.
10. UNCLASSIFIED
UNCLASSIFIED
10
Looking Back at Our Completed
Research Projects: 2006-2010
Know the Research Base Assess Grand
Challenges in “Using”
It
Apply “It” (WMD-Focus)
Social Dimensions of
Proliferation
(Sandia National Lab, 2007)
Explore the literature base on
terrorist WMD motivations.
Map points of divergence
and convergence in the
viewpoints.
Risks and Benefits of WMD
Activities for Non-State Actors
(Penn State University, 2008)
Explore the psychological
literature on terrorism and
potential relevance to WMD.
Assess social science
methodologies to
understand how people
perceive risk and benefits
of certain actions.
Consider which
methodologies would be
useful to a WMD terrorism
analyst and how they
might be best used.
Next Generation of WMD
Terrorism
(SAIC, 2008)
Assess current and future
WMD/WME terrorism trends
and the capabilities and
strategies needed to deal
with them.
Terrorist Innovations in
Weapons of Mass Effect
(Naval Postgraduate School, 2010)
Consider Western cases of
terrorists using WME.
Apply knowledge from case
analyses to identify
predicative indicators of
such activity.
Comparative Strategic
Cultures
(SAIC, 2008)
Understand the
challenges associated
with strategic culture
analysis.
Develop case studies of
how cultural norms impact
WMD decisions.
11. UNCLASSIFIED
UNCLASSIFIED
11
Looking Back at Our Completed
Research Projects: 2006-2010
Know the Research Base Assess Grand
Challenges in “Using” It
Apply “It” (WMD-Focus)
A Decision Framework for
Assessing WMD Threats Using
Social Agent Models
(Argonne National Lab, 2008)
Consider which elements
of computational modeling,
social science theory, and
decision theory might be
relevant.
Offer an approach to integrate
those contributions that could
be used to understand WMD
decisions.
Analytic Framework for
Tailored Deterrence
(SAIC/Penn State University, 2009)
Identify the relevant social
sciences research base on
tailored deterrence.
Identify the challenge in
using this research in
operational environments.
Develop an initial framework to
apply the knowledge to assess
tailored deterrence strategies.
What is a computational social
model anyway?
(Galisteo Consulting Group, 2008)
Explicate the modeling
process and identify
challenges and explore
limits and potentials of
using models.
Integrated Adversarial Social
Network Theory
(University of Kentucky, 2010)
Identify the theories in the
social science that can
shed insight on social
networks.
Assess ways to integrate
those theories together
and ways that “theory of
theories” might be
constructed.
Identify possible applications
of the “theory of theories” to
network assessments.
Guesswork: The Troubled
Past of Prediction
(Naval Postgraduate School, 2010)
Consider cases in history
where prediction of foreign
enemies failed.
Consider lessons learned
from those approaches
and relevancy for today.
12. UNCLASSIFIED
UNCLASSIFIED
12
A Look Back at 2006-2010:
Lessons Learned
• On WMD/E terrorism issues:
• Emerging area of research in the social sciences, but
still many gaps.
• Case study research may have broader value.
• Potential focus for modeling?
13. UNCLASSIFIED
UNCLASSIFIED
13
A Look Back at 2006-2010:
Lessons Learned
• On “grand challenges”:
• Prediction of adversarial behavior is hard, but this is
not a new challenge for the United States.
• “Old” ways of analysis may be valuable.
• Multi-method approaches work best.
• Don’t rule out qualitative approaches.
• Computational social models may have “creative” value for
analysis, but their predictive value may be less.
14. UNCLASSIFIED
UNCLASSIFIED
14
Our Focus Areas for 2010-2011
• Grand Challenges in Computational Social Modeling.
• Interdisciplinary, multi-expert approach to:
• Examine state of the art.
• Identify challenges for the field as it moves forward.
• Identify a set of principles to follow when funding, developing, and/or
using models in high-stakes decision making environments.
• Focus:
• Identify lessons from other fields and relevancy.
• Challenges: data, ethics, organizational culture, model assessment,
conveying results and ideas of what a model “can do.”
• Approach: literature review, expert discussions, white papers.
• Partner: SNL
15. UNCLASSIFIED
UNCLASSIFIED
15
Our Focus Areas for 2010-2011
• Think piece (Galisteo Consulting Group):
• Assessing Computational Social Model
“Goodness”
• If it is assumed that the value of such a models lies in
areas other than prediction, how can one assess
whether a model is good?
• What is a candidate methodology to assess them if
traditional validation is not feasible?
16. UNCLASSIFIED
UNCLASSIFIED
16
Broad Lessons Learned on Doing
Modeling Research
• Leverage experts in all fields, not just one.
• When doing “challenges” research, don’t just involve the
traditional community of modelers who do DoD research.
• Interdisciplinary research is key, but hard.
• Evolving focus is necessary and possibly a return to
fundamentals.
• Do what is needed, not necessarily what is “trendy.”
• One program can’t “do” everything.
• More projects, more funding, and more technology is not always
the path to scientific and application-based advances.
Editor's Notes
Hi, my name is Jennifer Perry and I lead a small social sciences research program in DTRA’s internal strategic studies office. the Advanced Systems and Concepts Office. This briefing is going to provide an overview of our past and current research agenda.
I’d like to accomplish a few things with this briefing. I will provide a brief overview of my office and this research program specifically and then discuss why we are doing a retrospective look at this program. I will then discuss, in more depth, the research we’ve accomplished over the last nine or so years. I will also identify some lessons learned that may be applicable to other government research programs in this area.
As noted previously, my office is called the Advanced Systems and Concepts Office. It essentially acts as a future or strategic studies office and we do research in a wide variety of scientific and policy disciplines. The social sciences research program I am going to discuss is but one tiny facet of this office. As we are located within DTRA, every research portfolio or project is focused on analyzing and addressing a specific type of national security threat, those related to weapons of mass destruction. However, we define WMD quite broadly. While some of our projects are done in-house with office staff serving as principal investigators as well as project managers, many of them are conducted through partnering with national labs, commercial contractors, and universities, as well as other think tanks or research centers. Most of our studies result in a publically releasable analytic report.
The social sciences research one is pretty strategically-oriented and we attempt to build on research conducted in previous years. However, what distinguishes this program from others in our office is that the program does include some projects that do not have an explicit WMD focus. For example, we’ve done a lot of work on the grand challenges of computational social modeling. One of the reasons for this is many in DTRA and beyond are interested in the WMD analytic application of such models. We felt that there was some utility in focusing on the limits, potentials, and challenges associated with such model development and application to national security threats at a broader level. This would increase awareness among the funder, user, and research community that works on WMD issues, but also other national security ones. In this way, the potential benefit of the research would be broader.
Like every research area in ASCO, the social sciences research agenda is not requirements-driven. In many cases, we do respond to needs others in DTRA and the broader government have identified. In other cases, ideas for projects are generated within the office or within our research partner base. In every case, we do something because we think it is a good idea and would have broad impact on the way in which WMD and related issues are understood, studied, and addressed. Because our budget is quite small, we try to select a small number of projects each year which are likely to have the most impact and benefit to the government and research communities.
One of the main reasons this briefing offers a mostly retrospective look at this research program is because ASCO, the office in which this program resides, is being eliminated in September 2011 as a result of the Secretary of Defense-mandated efficiencies. While other areas of DTRA may continue to undertake social sciences and modeling efforts, this program, in its current form, will not continue to exist. Due to it’s niche focus on grand challenges in modeling and WMD, this program is unique among the many in existence in the national security community. There may be lessons to learn and ways this research can be leveraged by others in the community in the future. For this reason, I offer this retrospective.
In its early years, our social science research agenda was known as the “Threat Anticipation Project” or TAP. Then DTRA director, Dr. Steven Younger, was interested in using computational social science models to anticipate threats such as 9/11 so we primarily focused on developing computational social science models to understand how they could be applied to a broad set of national security concerns. As part of TAP, ASCO guided the development of several prototype modeling efforts. These efforts did not focus only on WMD threat applications because we were encouraged to have a broad focus.
The neural net effort used a traditional Bayesian approach to see if it was possible to predict the behavior of historical leaders using personality profiles. It was an interesting exercise, albeit with a small sample, but it did not produce any new insights and lacked depth. During that same time period, we developed an expected utility model to look at the leadership interactions within al Qaeda on two issues- evolving views on target selection and choices of weapons to advance the cause. All analysis was based on unclassified data.
During this time, ASCO also funded two other significant prototype modeling efforts. One of them was the Cultural Simulation Model, which is a novel approach to allow a user to mine and analyze data and see it from different cultural perspectives. In its 3rd year, we asked the creators, Indasea, a small firm out in Maui, to apply the model to the case of Indonesia to demonstrate its potential. We are no longer funding this model, but other DoD research organizations have funded its further advancement.
Our other effort was conducted by Ed Mackerrow out of Los Alamos National Laboratory. His approach leverages traditional agent-based modeling and systems dynamics approaches and allows the user to create scenarios and see them played out by agents to better understand Islamist political violence. The agents are endowed with different properties and behaviors and interact on a GIS system. Various forms of his model are currently in use by several members of the community to assess real world situations and an unclassified analytic paper on his lessons learned through this initial experience is available.
We also funded the development of a joint project between the University of Chicago and Argonne National Laboratory, which would allow social scientists, both grad students and faculty, to work with modelers at Argonne and develop research projects to encourage this kind of sustained collaboration. This was called the Joint Threat Anticipation Center. Many of the research efforts developed focused on terrorism issues. In its later years, the project also included a series of working sessions between social science students and modelers to discuss some of the challenges in modeling identity, culture, and collective discourse, as well as some work on understanding the literature on computational model validation and how the ideas contained therein might apply or not to social science model validation. While this project is no longer active for a variety of reasons, it did suggest the value of this kind of sustained interaction and pointed to several challenges involved in developing this kind of initiative.
After we began our TAP project in 2002, we realized that many more DoD offices were getting into the modeling game. In 2006, we began to carve out a niche focus for us to do research that wasn’t being done elsewhere. We decided to focus more on WMD and on grand challenges issues in modeling. Our WMD research would use a wide variety of social science knowledge, theories, and approaches. Not every project would be a modeling one. We would focus on three main areas or bins: understand the existing research base which pertains to a problem set or question of interest, assess the grand challenges in using that research at an applied level, and apply the insights to WMD-focused analysis. In many cases, a single project can have focus points which span the 3 bins.
The grand challenges bin, which is likely of most interest here, includes our work on the limits and potentials of computational social modeling for decision or analytic support in operational, high-stakes environments.
Our research during this time frame had several themes: it included a heavy focus on WMD/E terrorism issues and a focus on uncovering the limits and potentials of applying research and scientific advances in several areas of major interest for the USG to both understanding and analyzing national security threats at a broad level, and WMD threats more specifically.
Although we did develop one study that had a state-based focus, we primarily concentrated our efforts on understanding the evolving nature of the WMD threats posed by non-state actors and terrorist groups specifically. This research involved a lot of case studies, but also literature reviews, and policy analysis efforts to identify ways to combat the threat over the long-term that were grounded in social science research in the academic arena. Overall, this area of study is not a major focus in the social science research communities, but there is a growing attention and knowledge base on this problem. Much of our research focused on arguing that it is insufficient to focus either on capabilities/feasibility or motivations and intentions when developing analyses to assess the WMD threat potential associated with one specific terrorist group or in general. It is more appropriate to consider both in tandem with one another. It can be challenging to apply academic research on this issue in an operational environment, but there are several ways in which the academic research base can be a valuable source of analysis for the operational and decision making communities in the USG.
Beyond our WMD focus, we also commissioned research projects which encouraged a “return to fundamentals.” For example, the idea of using strategic culture or tailored analysis has gotten a lot of traction and attention in DoD and affiliated research communities. Likewise, social network analysis and computational social modeling. The idea of predicting adversarial behavior is also a hot topic. In most cases, our research in these areas sought not to apply these approaches to a specific problem set, but to expose the major challenges and potentials for using them to better understand our adversaries. This would be one method of advancing the dialogue about these approaches within the research and government communities, but also promote mutual awareness of the issues and provide building blocks for future applied, problem-specific research.
I’d like to briefly discuss those projects which we’ve completed since our program’s restructuring in 2006. It is not the purpose of this briefing to provide a complete overview of the goals and findings of each research project, but to briefly discuss what we did for each of them in a sentence or two. If you have specific interest in one or more of these projects, I encourage you to discuss them with me after this briefing. I can also provide you with copies of the study reports.
We conducted four research projects on WMD/E terrorism issues though each had a different focus. Social dimensions of proliferation, which was done by Nancy Hayden at Sandia National Labs, was essentially a literature review to determine what research existed in the academic, industry, and government communities on WMD terrorism motivations. It was to be a baseline for future research and map the points of convergence and divergence of viewpoints on these issues in the literature using both human and software analytic capabilities. She found that it is primarily political scientists working on these issues and there are many gaps, though this is a growing research area. As if 2007, she found that there were not many peer-reviewed pieces that focus specifically on WMD terrorism motivations, but there is a larger selection of articles in related areas such as radicalization and violence, which may offer relevant insights. This literature base, she argued is underutilized.
Around the same time, we commissioned a similar study from Kevin Murphy, an organizational psychologist at Penn State. He focused on doing a literature review of psychology of terrorism literature and assessing the research which is most relevant to understanding WMD terrorism, specifically. This review would provide the foundation for a phase II study, which would consider the analytic approaches in the social sciences, but primarily psychology, that could be used to assess and understand how a terrorist might perceive the risk and benefit of acquiring and/or using WMD to achieve a goal. He assessed a variety of approaches, mostly those grounded in rational choice theory and offered an analysis of the limits and potentials of each of those approaches to be used in operational environments where an analyst is required to study WMD situations involving non-state actors, but mostly terrorists. He found that when considering criteria such as relevance, robustness, and information requirements, no one approach would be sufficient, but a combination of approaches might prove useful.
The next project on this list involved a variety of terrorism experts in academia, government, and industry, but the principal investigator was Lew Dunn at SAIC. The Next Generation of WMD Terrorism project moved beyond the literature review phase and sought to develop original analyses and use those analyses to identify ways to combat the threat. It sought to apply social science and related knowledge to better understand current and future trends in weapons of mass destruction and weapons of mass effect terrorism. This study assumes that while the threat from Al Qaeda “the core” appears diminished, the threat posed by extremist non-state actors has proliferated. Current trends indicate that this next generation of terrorism, sometimes referred to as Al Qaeda 2.0, will likely be more diffuse, decentralized and unpredictable than its predecessor. We commissioned papers and conducted related workshops and expert discussions to better understand these trends, from motivation and intent, to capacity, and identify strategies and capabilities needed to deal with the threat today and in the next 15 years. Some areas of emphasis included identifying ways to influence these actors, strategies and activities to identify and reduce US vulnerabilities for attacks, and better understanding terrorist campaign strategies.
The next and final project that focused on terrorism, was recently completed. Mohammed Hafez and Maria Rasmussen at the Naval Postgraduate School served as principal investigators for this study, but they involved a wide variety of high profile terrorism experts. Different from our previous studies, this one had a broader lens. It would consider not only situations of terrorists using weapons of mass destruction, as traditionally defined, but also weapons of mass effect. The planes that were used on September 11 are examples of weapons of mass effect. In August 2010, these experts gathered in Monterey, CA for a workshop to discuss the preconditions, causes, and predictive indicators associated with terrorist innovation in weapons of mass effect (WMEs). They presented their research findings on seven Western historical and contemporary cases of terrorist innovation, ranging from airplane hijackings by the Popular Front for the Liberation of Palestine (PFLP) to the current threat emanating from Al-Qaeda's mass casualty attacks. These case studies generated a number of generalizations about what motivates innovation, how terrorists come to innovate, and whether it is possible to anticipate innovations in WME terrorism. We note there is a possibility to extend this analytic focus to include Eastern cases to determine if the generalizations still hold and offer additional insights.
The final project on this list had a WMD focus, but it mostly concentrated on state-based issues. It was also our first attempt to return to fundamentals and assess the value of a comparative strategic culture approach to provide insights on our adversaries and friends, in particular, help us understand how they perceive WMD and how they make decisions related to it. We developed a series of commissioned papers from country and strategic culture experts to identify the areas of possible application of this analytical tool to inform WMD understanding, including the development of case studies on Iran, Israel, Pakistan, China, Russia, the United States, India, North Korea and al Qaeda. There was a general consensus among the paper authors about the utility of a strategic culture approach in examining behavior and actions of foreign entities. This approach is gaining recognition within the national security community, but there is a need for caution when applying it, due to ongoing definitional and methodological concerns. With respect to using this approach to understand decisions by foreign adversaries, most of the country case study authors agreed that strategic culture, influenced by historical experiences, geography, and other factors (such as religion) provides a guiding force in how actors approach decisions within the international system, including those involving WMD. Although a common conclusion about “strategic cultures” of actors cannot be reached, an over-generalized common theme is that each culture, fundamentally, provides the actor for a means to rationalize decisions, behavior, and actions, and serves a function to reinforce views about their place within the international system. Notions of superiority, entitlement, and importance abound in almost all of the cases. The analyses from this project also served as the basis for a non-DTRA sponsored book, which was edited by several members of the project team.
We also developed two studies which sought to apply social science-based analytic techniques to WMD problems. Those studies are “A Decision Framework for Assessing WMD Threats Using Social Agent Models” and “Analytic Framework for Tailored Deterrence.” In both of these cases, we viewed this research as applied, but still fundamental in nature. That is, we did not completely develop such analytic frameworks, but rather considered what candidate ones might look like if they were to be developed. Both studies are foundations upon which others might build with further applied, problem-specific research. Both studies focus on the potential application of analytic techniques receiving a lot of attention in the DoD to better understand and address WMD problems and less on grand challenges associated with using these techniques, which is the focus of some of our other studies that I will discuss in a few minutes.
The former study sought to consider how computational social modeling, social science theory, and decision theory might be collectively used to better understand WMD decisions that our adversaries or others might undertake. David Sallach, a modeler and sociologist at Argonne National Laboratory analyzed how theories in social science, decision science, along with computational social modeling could be integrated together to form an analytic framework to better understand complex WMD decisions about risk. He advocates for a social agent model approach to a decision framework and argues that all three sciences have something necessary to contribute to formulating it.
The latter study on tailored deterrence actually was attempted by two research organizations- SAIC and Penn State, though each involved a variety of deterrence experts. In some cases, these expert pools overlapped. The assumption for both of these studies was that some would argue that developing a tailored understanding of how an adversary behaves, acts, and what motivates them, is crucial to ensuring successful deterrence, both at a strategies and implementation level. Although much has been said about the need to tailor deterrence, we observed that there exists no in-depth study on the contributions of the social science literature base to inform this practice. As such, he goals of each of the studies were to identify the existing social science research base on tailored deterrence, including assessments of how it might be used in real world environments and the limits and potentials associated with the approach. Workshops, involving experts, would then be convened to make a cohesive list of the challenge of using social science research and analytic approaches to tailor deterrence strategies for a specific adversary. They would then discuss how these challenges might be overcome and ways to best facilitate the application of the social science research and approaches in real world situations involving decision makers needing to either develop a strategy or assess the courses of action associated with one. These projects turned out to be more ambitious than we first realized. Overall, many challenges were identified, but other research will be required to figure out ways to deal with them. However, both studies are potential starting points for that kind of investigation.
The last three studies on this list have a broader focus on grand challenges and extend beyond the WMD domain. They advocate for a return to fundamentals, an assessment of current analytic approaches receiving a lot of interest in the DoD, including computational social modeling, social network analysis, and qualitative and quantitiative adversarial prediction. In most cases, they seek to explore the limits, potentials, and/or needed advancements in these areas to allow them to be fully and responsibly leveraged to aid in national security threat analysis and decision making in real world situations.
The first project in this area is “What is a Computational Social Model Anyway?” One of the reasons ASCO has gotten out of the model development game is because many others in the community have now forged ahead with developing models and focusing heavily on improving the technology. This is a costly endeavor and ASCO thought its resources would be better placed against something that was not receiving as much attention. ASCO saw a need to take a step back and consider the challenges of applying these models to national security problems. Interest is very high but knowledge about this approach’s limitation is not generally high within the user community. Many have unrealistic expectations about what models are and what they can do. One specific concern is not only ensuring models, if they are going to be used, are being used against the right kinds of questions, but also understanding the ways in which the model development process, and the shortcomings associated with it, impact how models can/should be used. Jessica Turnley, a cultural anthropologist at Galisteo Consulting Group Inc., did a study for us that examined these issues. Rather than focusing on technical challenges associated with model development, Jessica primarily focuses on some of the softer issues involving social science data and modeling team development roles, and discusses how these challenges can limit the way models are used. She does not advocate using models for prediction but as a means of stimulating creative thinking, deal with large sets of data, or replicate problem-solving processes. A computational social model, she suggests, should be viewed in analogic terms in that some can identify that which is important, unimportant, or not yet known to be important or unimportant in the target domain of analysis.
The second project, “Integrated Adversarial Network Theory” was developed as part of a larger agency-wide basic research activity. The particular study was, however, sponsored and overseen by ASCO. We began this activity because many USG research projects are underway to improve SNA methodology, such as finding ways to do the analysis faster and/or handle more data, or handle larger networks. Though this “technology-based” research is necessary, it is only one piece of the SNA puzzle and does not address other fundamental research challenges in the social science domain. Although social network analysis is focused on understanding a social network’s structure, that is, who is connected to whom and how, it is generally acknowledged that social and cultural influences underlie these connections and shape the network and its activities. These influences are unobservable, yet they are key to understanding how and why social networks form and change, what motivates the collective activities of actors within them, and how those activities are executed and for what purpose. Despite the centrality of these influences, little research has been done to examine what the social science theoretical contributions are to understand them across all fields of study and how they can be integrated. Steve Borgatti from the University of Kentucky therefore developed a 3 year research study to help address this research gap. He sought to uncover the fundamental principles from each social science discipline and figure out how to integrate them together in an overarching set of principles that might inform broad understanding about social networks, including the underlying social and cultural influences on those networks. He identified the challenges and values in doing this and offered ideas for future research.
The last project on this list, “Guesswork: The Troubled Past of Prediction” was our attempt to return to basics on this issue of “can adversarial behavior be predicted and what is the best way to do it and what are the limits involved?” We realized there was a lot of interest in the community on the predictive capability of computational social models today so we commissioned Zach Shore, a historian at the Naval Postgraduate School to develop an analysis of the “hot approaches” to prediction the United States has used in the 20th century. He developed several cases of situations in which the United States sought to predict its enemies behavior through qualitative and quantitative means and identified lessons learned from selected cases in which our prediction failed. He would also consider the relevancy of those lessons learned for today and offer an assessment on the benefits associated with qualitative prediction within the context of our increased infatuation with other predictive approaches, including computational social modeling. Cautioning against any approach which substitutes technology for human analytic judgment, the paper concludes that consistently accurate prediction of individual behavior in foreign affairs is highly unlikely because individual motives are so often opaque. Yet it also suggests that thoughtful, non-quantitative analysis can produce reasonably accurate predictions of group behavior. The paper is less optimistic about quantitative or computational modeling approaches to prediction, though the latter is only given a cursory treatment.
Several lessons were learned from our studies during this era. I will briefly outline a few of them. Regarding WMD/E terrorism analysis, this is an emerging area of research that should continue to receive attention in the social sciences research community as many gaps in the research exist. We encourage researchers and program managers to identify ways case study methodologies might be employed to fully understand how several factors, namely motivations, intentions, capabilities, and feasibility, interact with one another and inform situations in which a terrorist group might consider or not consider acquiring and using WMD. Modeling approaches might also be used to conduct this analysis, but any project that uses it, will need to explicitly outline the assumptions, limits, and potentials associated with that approach relative to the specific question or problem being addressed. Within our program, we were able to explore the value of qualitative research, but computational social modeling approaches were, for the most part, not used.
To recap, our program also included projects which did not focus on just WMD issues, but also grand challenges associated with using social science knowledge to assess, and even anticipate, national security threats. Overall, we found that, well, prediction of adversarial behavior is hard. But it has always been hard. The challenges we are facing in this post-September 11 era may be slightly different, but they are not all new. We should learn from the lessons of the past when devising and using particular approaches to predict adversarial behavior. Predictive analysis is, in and of itself, not a useless endeavor, but expectations for what might be achieved need to be realistic. When offering the outcomes of an analysis to a decision maker, one should be frank in the limits of the analysis. Overall, one should probably consider using multiple approaches, and qualitative approaches in particular should not be ruled out in favor of the flashy technological approaches. However, all approaches have limits, including qualitative ones. Computational social modeling approaches have limits, but potentials also exist in providing creative input to an analyst and offering potentials that he/she might have not thought of on his/her own. However, it would be a mistake to suggest that these notions apply to every predictive situation. The individual factors at play vary from context to context.
Before I end this briefing, I wanted to briefly outline ongoing and “soon to be started” efforts. Given the situation with our office being eliminated, we currently only have one project that is ongoing in this area. However, it is a major one. It is a cornerstone upon which others can build and can also be seen as a culminating effort to assess the grand challenges of computational social modeling and in particular, challenges in developing and using these models to support decision making in real world environments.
Laura McNamara, an organizational anthropologist, and Tim Trucano, a mathematical physicist at Sandia National Laboratory are leading a project for us to identify the state of the art in computational social modeling, identify sets of challenges the computational social modeling community needs to address as it moves forward and how lessons learned from other scientific disciplines might apply, and from there, identify a set of principles which might frame how models, intended to provide decision support in national security organizations, are funded, developed, and used. Realizing this is an inherently interdisciplinary research question, the analysis involves a group of experts that span many scientific disciplines. In early to mid 2010, Laura and Tim developed an analysis which examines these questions. They then asked a group of experts to expand on that analysis and devise analytic white papers on one particular challenge area. These papers focused on such varied issues as data availability and use, ethics associated with developing and using models, organizational culture and its impact on how models are used, and more fundamentally, viewed, approaches to assess model “goodness,” and conveying model “outputs” to decision makers. The distinguished group presented and discussed their papers at a workshop in October 2010. Those papers are currently being finalized and a workshop report is being drafted. Next steps may include a DC-based seminar to convey the results of this study and allow for further discussion among the model funder, developer, and user communities on the issues discussed in the study.
A final report is expected in the summer of 2011. We intend to subject it to public release approval.
The last project I am going to discuss, and the last one that is funded under this program, is a think piece which extends our previous and ongoing work on the grand challenges of computational social modeling. Jessica Turnley, author of the study I previously discussed entitled “what is a computational social model anyway?” will develop an analysis which offers one or more candidate approaches for assessing whether a model is good if the goal of that model is not prediction. She will explore what assessment approaches might be used if it is assumed that traditional validation, such as is done with models of physical phenomenon, is not possible when the models are of social phenomenon. We hope this, like our other project with Sandia National Lab, will be a starting point for future research undertaken by others in the government in this area.
Thanks for listening to this briefing. I hope this largely retrospective discussion proved interesting and relevant to your areas of interest. I’d like to leave you with two lessons we learned in this program about doing computational social modeling research. We’ve all heard that this is an inter-disciplinary research area. That’s true. I’d like to propose that using experts in all relevant fields is imperative for any research project in this arena, but particularly important when the research is fundamental in nature and seeks to address broad challenges, or at least identify them. Experts need to be drawn from a wide variety of disciplines and we need to remember that social scientists are a diverse group of experts. They represent a variety of disciplines, but no two social scientists in one discipline are exactly alike. Every effort should be made to involve the expert that most makes sense to involve in a given project. However, even this is not sufficient. Particularly when the research is “challenge-based” like most of our research has been, one needs to involve the usual suspects that are involved in DoD modeling activities, but also look beyond that community to not only other research communities involved in computational social modeling, but also communities that aren’t explicitly in the social sciences or modeling game. Doing so can be difficult and you may make some missteps along the way, but if the pool of experts that one uses in a project is limited to the known entities, one is not likely to achieve major scientific advances or get unique insights that may be applicable to a problem set.
We recognize that there is a real need within the DoD to do modeling R&D at an applied and basic research level and do not suggest that the research in this area be abandoned for the kinds of challenges-based research I discussed. Both have their place and are needed. However, in every way, research agendas should be based on what needs to be done and not necessarily what is trendy. One program can’t and shouldn’t do everything if depth is desired. Having a niche is not always a bad thing. Overlapping research can be good, but it can also hinder advancement for the broader good. More projects, more funding, and more budgets aren’t necessarily what is needed to achieve scientific advancement and impact, but they can’t hurt. However, we should be careful to avoid an over-reliance on technology to solve our problems.
Thanks again for your attention and I look forward to any questions you might have.
