The document summarizes the evolution of a team's project from helping analysts automate the time-consuming process of drawing geospatial boundaries around protected sites, to developing voice transcription tools to help pilots and ground troops communicate more effectively during time-sensitive military operations, in order to minimize civilian casualties. The team interviewed experts from the military, intelligence community, tech companies, and NGOs to understand the challenges and shifted their focus based on feedback to address the problem of potential mischaracterization of targets during dynamic strikes requiring quick decision-making. They are now building prototypes to streamline radio communications between pilots and ground troops to reduce cognitive load and improve confidence in targeting.
1. Targeting analysts need to
develop effective tools to aid in
the identification of protected
sites in order to preserve them
during military operations.
Sponsor
Problem
Statement
:
Pilots miss key ground-level
communications from JTAC’s in
time-sensitive situations, increasing
the risk of civilian casualties
(CIVCAS).
Current
Problem
Statement
:
100 Interviews
to date
Michael Cooper
BS CS, MS CS
Jason Ginsberg
BS EE, MS CS
Xinlan Emily Hu
BS CS, MS SymSys
Matthew Trost
BS CS, EE Minor
Sponsor:
2. Article 54(2) of the 1977 Additional Protocol I provides:
It is prohibited to attack, destroy,
remove, or render useless objects
indispensable to the survival of
the civilian population...
Overarching Motivation: Protect Civilian Lives
4. We interviewed 100 people:
Combat Forces
Fighter Pilots
JTACs
Targeteers
Ground Commanders
Special Forces
AI Engineers
Apple, Facebook,
Microsoft, Google
NGOs
Intelligence Community
Fmr. Directors: NSA, CIA, NRO
Analysts: NGA, DIA
Combatant Commands
AFRICOM, INDOPACOM, STRATCOM,
CENTCOM, SOCOM, EUCOM
Deployment
Prime Defense Contractors
Air Force Research Laboratory
6. Starting with our Sponsor’s Problem
WEEKS 1-2
Geospatially defining No
Strike Facilities is time
consuming for analysts.
No Strike Boundary
7. Saving Analysts’ Time with AI
Entries on No-Strike Lists lack “geospatial boundaries”.
● Boundaries are useful for targeting and weaponeering.
● Analysts draw by hand -- incredibly slow.
An AI solution to this problem would save years of analyst time.
- Combatant Command Chief of Targets
8. Taking a step back: learning about targeting process.
Planning Stage
Commander’s
Intent
Courses
of
Action
Weaponeering
Battle
Damage
Assessment
Air Tasking
Cycle
Patrol ID
Intelligence
Intelligence
Intelligence
3 Days
Targeteering
9. Drawing boundaries is just one small piece.
Planning Stage
Commander’s
Intent
Courses
of
Action
Weaponeering
Battle
Damage
Assessment
Air Tasking
Cycle
Patrol ID
Intelligence
Intelligence
Intelligence
3 Days
Targeteering
10. (1) Intelligence Supports Decision Making
Planning Stage
Commander’s
Intent
Courses
of
Action
Weaponeering
Battle
Damage
Assessment
Air Tasking
Cycle
Patrol ID
Intelligence
Intelligence
3 Days
Targeteering
Intelligence
11. NGO Self-
Reports
MIDB
Contains info on targets & NSFs
Used by targeteers
SIGINT HUMINT GEOINT MASINT
Targeteers get Intel from the Modernized Integrated DB
12. Patrol ID
(2) Dynamic vs Pre-planned Strikes
Intelligence
Planning Stage
Commander’s
Intent
Courses
of
Action
Weaponeering
Battle
Damage
Assessment
Air Tasking
Cycle
Intelligence
Intelligence
3 Days
Targeteerin
g
13. Shifting Focus to Dynamic Strikes
WEEKS 2-5
Dynamic strikes have higher risk
of mischaracterizing targets.
14. Risk of Target Mischaracterization in Dynamic Strikes
Apartment
Building
Mosque
Enemy HQ
15. Risk of Target Mischaracterization in Dynamic Strikes
Apartment
Building
Mosque
Enemy HQ
Planting IED? Irrigation
16. If you want to protect humans, you have to
understand the local civilian patterns of life.
- UN Civilian Protection Specialist
21. Reactions were lukewarm.
“Unless you could look inside walls…”
“How do you trust that the info
isn’t being manipulated?”
“That’s interesting but not compelling.”
22. Getting the Data is Hard
In my year of running targeting in Iraq, we saw maybe four incidents.
– Lead Intel Targeteer, Iraq
vs.
29. KEY PARTNERS
● Air Force
Research
Laboratory
● 13th ASOS /
92nd AGOW
● JAIC
● 422nd Test &
Evaluation
Squadron
KEY RESOURCES
● AI expertise to re-
train SOTA models
on domain specific
data set.
● 1,000 hrs of
transcribed JTAC
training data
VALUE PROPOSITIONS
Streamline JTAC/pilot
comms. during CAS
missions.
Reduce pilots repeat-
requests from the JTAC.
Increase confidence in
informational relay,
correlation, and targeting:
pilots, JTAC, civilians, all
find this valuable.
MISSION ACHIEVEMENT/IMPACT FACTORS
Reducing cognitive load on Pilots, so they can better process ROE.
JTACs require less time to talk pilots on to a target via radio. Yet, still
retain digital efficiencies.
KEY ACTIVITIES
● Implement ASR for
pilot-JTAC comms
● Design simple UX
with replay, aut-
parsing, speaker
notation, and
customization. DEPLOYMENT
Construct voice-to-text
system running on pilot
tablets. Link radio stream
into tablet auxiliary input.
On-device inference.
BUY-IN & SUPPORT
Work within key partners
to integrate this product
into the ATAK system.
MISSION BUDGET/COST
Maximize spending on data and engineers.
Burn rate of $60,000/mo.
BENEFICIARIES
JTAC
USAF Fighter Pilots
Civilians
KEY PARTNERS
● Air Force
Research
Laboratory
● 13th ASOS /
92nd AGOW
● JAIC
● 422nd Test &
Evaluation
Squadron
31. Looking Back
Where we started:
helping analysts save time
Where we are now:
helping pilots save civilian lives
32. Looking Back
Where we started:
helping analysts save time
Where we are now:
helping pilots save civilian lives
Help super multi-taskers
33. Looking Back
Where we started:
helping analysts save time
Where we are now:
helping pilots save civilian lives
Help super multi-taskers focus on
the tasks that matter the most.
34. Huge Thank You to:
Our Mentors:
Rafi Holtzman, Jennifer Smith-Heys, Matthew ‘Fuego’ Tuzel
Our Sponsors:
Joe Murray, Matt Moore
The Hacking for Defense Course Staff:
Nate Simon, Sam Lisbonne, Valeria Rincon, Steve Blank, Steve
Weinstein, Joe Felter, Pete Newell, Tom Bedecarré, Jeff Decker
And Finally:
Our classmates, and especially all of our interviewees.
Hello, we are Team Protocol One, sponsored by the Joint Artificial Intelligence Center.
Our overarching motivation was to protect civilian lives.
Indeed, our name, Protocol One, is inspired by the Additional Protocol One to the Geneva Conventions, which prohibits harming civilians.
Civilian Casualties, or CIVCAS is not only a moral and legal imperative to be concerned about. It is also a strategic priority.
Civilian death leads to more people being driven to anti-U.S. forces, which in turn works against our objectives.
It’s important to mention, however, that the vast majority of CIVCAS in conflict is not caused by US forces. The US is very attentive to protecting civilians. However, the US does make mistakes.
For example, in addition to multiple smaller incidents, the two most notorious are:
The accidental strikes of the Chinese Embassy in Belgrade in 1999
and of a Doctors without Borders in hospital in 2015.
To understand why these accidents happen, we have interviewed 100 people, ranging from Intelligence Professionals to Pilots to experts from NGO, military, and Artificial Intelligence communities.
This presentation will cover how we approached the problem of civilian casualties over the past 10 weeks.
When we first entered the class, we were given a very specific problem statement:
That geospatially defining No-Strike Facilities is time-consuming for analysts. This slide shows an example of a no-strike facility (specifically, a school), surrounded by a box defining its boundaries.
Why are these boxes important?
It turns out that knowing exactly how big the no-strike facility is helpful for targeting and weaponeering. Currently, analysts do this by hand, which is far too slow.
The solution that was originally on our minds was to do this via an AI, which would turn an unbounded satellite image (as you see at left), to the bounded image at right.
But we wanted to step back into the larger process of targeting, because our solution shouldn’t just be about saving analysts time. We wanted to know how this would impact civilian casualties.
By talking to targeteers and looking at the lengthy Joint Targeting Manual, we pieced together this high level understanding of the Joint Targeting Cycle.
We learned that the drawing of boundaries was a very small part in a very large process.
Specifically, we gained two key insights.
First, we learned that, throughout the process, intelligence supports decision making -- as you can see in these three highlighted areas.
Where does this information come from? The Modernized Integrated Database.
This is where the NSF Geospatial boundaries would be created, stored, and accessed before a strike.
Information can come from various sources, as you can see on this slide.
Our second key insight is that there is a difference between pre-planned strikes, which are typically done on fixed military targets that have gone through extensive analysis and vetting, and dynamic strikes, which are done on time-sensitive, potentially mobile, targets.
But we found that relative to dynamic strikes, CIVCAS was very rare in pre-planned strikes.
So although geospatially defining No Strike Facilities is an important part of the targeting process, we had difficulty connecting that task to downstream civilian harm. Analysts are able to draw the boundaries and that information is used.
Instead, we shifted our focus to dynamic strikes, which we found to be the most relevant to CIVCAS because they are time-constrained and can occur in potentially unfamiliar locations, so they have a higher risk of mischaracterizing a target.
For example, in war zones, buildings can often change functions due to the rapidly changing situations on the ground. A residential building may be used as a mosque if the mosque is under repair. Keeping up with these changes is a challenge.
Or consider the example where you’re flying overhead at 2am and you see people digging at the side of the road. You might assume they’re planting an IED and are a threat, but if you were familiar with that village, you’d know that the locals dig their irrigation trenches at night since it’s too hot during the day.
Our key takeaway from discussing these types of risks with pilots, troops, and targeteers was that understanding local civilian behavior can help prevent mistakenly targeting benign structures or activity.
This idea was corroborated by a civilian protection specialist at the UN, who told us that “If you want to protect...life”
This contrasts with traditional intelligence; although the MIDB has information on targets, it generally doesn’t describe the behavior of non-targets.
This brings us to our second MVP, a crowdsourcing platform and database for localized and up-to-date information on civilian activity.
Ground troops, local military partners, Open Source information (such as social media), and the civilians themselves would contribute to....
...a database of local population flows, cultural norms, and more...
...which is then vetted and refined by analysts...
...and made available to troops, pilots, and targeteers to provide potentially life saving civilian context for the missions they run.
This represents a major divergence from the intelligence inside of MIDB, which does not generally tap these sources and can only be curated by professional intelligence analysts, which limits how fast it can be updated.
Unfortunately, we found that people were lukewarm about this idea. They couldn’t point to concrete pain points in their workflows that our MVP would solve.
In fact, most of the people we spoke with had trouble connecting their direct work to downstream civilian casualty incidents in general.
Most were only familiar with a handful of incidents that were hard to generalize.
So although we found a DoD report listing dozens of incidents with hundreds of casualties, our interviewees didn’t know the context for those incidents. And we were ultimately unable to connect with anyone who did know.
We couldn’t take our theoretical MVP and map it to actual pain points that operators experienced, so our MVP development stagnated.
...and we pivoted once more.
This MVP was the first where our end users –pilots and JTACs– were truly excited by what we built.
The reason is that pilots no longer need to worry about perfectly copying everything down, even when the radio cuts out.
And JTACs can now quickly send digital nine-lines, using only their voice.
So now we’re looking to deploy our solution, with the help of partners across the air force and DoD.
This summer we’re looking to raise money through AFSOC, AFWERX , and H4X Labs, which we’ll use to hire more engineers and process data. Our data has already, been provided by The 93rd Air Ground Operations Wing.
A key to our success will also be working with the AFRL, who owns the tablet hardware, and the JAIC who can help us deploy our AI system. Finally, our beta testers will be the 13th air support ops squadron.
Look, we learned a lot this quarter but looking back, what it’s boils down to is quite simple.
We started by helping analysts save time. Which helped civilians downstream.
Now we’re focusing on saving civilian lives directly, by aiding the pilot at the final link in the chain.
The way we do this is by providing pilots with the data they need
so they can focus on the tasks, like rules of engagement, that matter the most.
The way we do this is by providing pilots with the data they need
so they can focus on the tasks, like rules of engagement, that matter the most.
We’re continuing this project, so if anyone in the audience can help us or knows someone with relevant experience, please email …
Thank you!