Protocol One H4D 2020 Lessons Learned
•Download as PPTX, PDF•
0 likes•53,747 views
The document summarizes the evolution of a team's project from helping analysts automate the time-consuming process of drawing geospatial boundaries around protected sites, to developing voice transcription tools to help pilots and ground troops communicate more effectively during time-sensitive military operations, in order to minimize civilian casualties. The team interviewed experts from the military, intelligence community, tech companies, and NGOs to understand the challenges and shifted their focus based on feedback to address the problem of potential mischaracterization of targets during dynamic strikes requiring quick decision-making. They are now building prototypes to streamline radio communications between pilots and ground troops to reduce cognitive load and improve confidence in targeting.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Protocol One H4D 2020 Lessons Learned
Similar to Protocol One H4D 2020 Lessons Learned (20)
More from Stanford University
More from Stanford University (20)
Recently uploaded
Recently uploaded (20)
Protocol One H4D 2020 Lessons Learned
- 1. Targeting analysts need to develop effective tools to aid in the identification of protected sites in order to preserve them during military operations. Sponsor Problem Statement : Pilots miss key ground-level communications from JTAC’s in time-sensitive situations, increasing the risk of civilian casualties (CIVCAS). Current Problem Statement : 100 Interviews to date Michael Cooper BS CS, MS CS Jason Ginsberg BS EE, MS CS Xinlan Emily Hu BS CS, MS SymSys Matthew Trost BS CS, EE Minor Sponsor:
- 2. Article 54(2) of the 1977 Additional Protocol I provides: It is prohibited to attack, destroy, remove, or render useless objects indispensable to the survival of the civilian population... Overarching Motivation: Protect Civilian Lives
- 3. Chinese Embassy Belgrade, Serbia, 1999 MSF Hospital Kunduz, Afghanistan, 2015
- 4. We interviewed 100 people: Combat Forces Fighter Pilots JTACs Targeteers Ground Commanders Special Forces AI Engineers Apple, Facebook, Microsoft, Google NGOs Intelligence Community Fmr. Directors: NSA, CIA, NRO Analysts: NGA, DIA Combatant Commands AFRICOM, INDOPACOM, STRATCOM, CENTCOM, SOCOM, EUCOM Deployment Prime Defense Contractors Air Force Research Laboratory
- 6. Starting with our Sponsor’s Problem WEEKS 1-2 Geospatially defining No Strike Facilities is time consuming for analysts. No Strike Boundary
- 7. Saving Analysts’ Time with AI Entries on No-Strike Lists lack “geospatial boundaries”. ● Boundaries are useful for targeting and weaponeering. ● Analysts draw by hand -- incredibly slow. An AI solution to this problem would save years of analyst time. - Combatant Command Chief of Targets
- 8. Taking a step back: learning about targeting process. Planning Stage Commander’s Intent Courses of Action Weaponeering Battle Damage Assessment Air Tasking Cycle Patrol ID Intelligence Intelligence Intelligence 3 Days Targeteering
- 9. Drawing boundaries is just one small piece. Planning Stage Commander’s Intent Courses of Action Weaponeering Battle Damage Assessment Air Tasking Cycle Patrol ID Intelligence Intelligence Intelligence 3 Days Targeteering
- 10. (1) Intelligence Supports Decision Making Planning Stage Commander’s Intent Courses of Action Weaponeering Battle Damage Assessment Air Tasking Cycle Patrol ID Intelligence Intelligence 3 Days Targeteering Intelligence
- 11. NGO Self- Reports MIDB Contains info on targets & NSFs Used by targeteers SIGINT HUMINT GEOINT MASINT Targeteers get Intel from the Modernized Integrated DB
- 12. Patrol ID (2) Dynamic vs Pre-planned Strikes Intelligence Planning Stage Commander’s Intent Courses of Action Weaponeering Battle Damage Assessment Air Tasking Cycle Intelligence Intelligence 3 Days Targeteerin g
- 13. Shifting Focus to Dynamic Strikes WEEKS 2-5 Dynamic strikes have higher risk of mischaracterizing targets.
- 14. Risk of Target Mischaracterization in Dynamic Strikes Apartment Building Mosque Enemy HQ
- 15. Risk of Target Mischaracterization in Dynamic Strikes Apartment Building Mosque Enemy HQ Planting IED? Irrigation
- 16. If you want to protect humans, you have to understand the local civilian patterns of life. - UN Civilian Protection Specialist
- 17. On-the-Ground Sources Next MVP: Local Civilian Patterns-of-Life
- 18. On-the-Ground Sources Local Life Database Next MVP: Local Civilian Patterns-of-Life
- 19. On-the-Ground Sources Local Life Database Next MVP: Local Civilian Patterns-of-Life Analysts
- 20. On-the-Ground Sources Local Life Database Next MVP: Local Civilian Patterns-of-Life Analysts On-Hand Access Commanders Ground Troops + Pilots Targeteers
- 21. Reactions were lukewarm. “Unless you could look inside walls…” “How do you trust that the info isn’t being manipulated?” “That’s interesting but not compelling.”
- 22. Getting the Data is Hard In my year of running targeting in Iraq, we saw maybe four incidents. – Lead Intel Targeteer, Iraq vs.
- 23. We reached an impasse.
- 24. Until one pilot presented a new problem in this space...
- 25. Until one pilot presented a new problem in this space... and we pivoted.
- 27. Protocol One’s Problem Evolution WEEKS 5-10 Pilot-JTAC communications demand a high cognitive load.
- 28. Key Beneficiaries Pilots PAINS Radio interrupts Hand-copying GAINS Replay + Redundancy JTACs PAINS Fear accidental friendly or civilian fire GAINS Greater confidence. One-and-done Civilians PAINS Disruption, threat to life, injury. GAINS Improved economic, social welfare
- 29. KEY PARTNERS ● Air Force Research Laboratory ● 13th ASOS / 92nd AGOW ● JAIC ● 422nd Test & Evaluation Squadron KEY RESOURCES ● AI expertise to re- train SOTA models on domain specific data set. ● 1,000 hrs of transcribed JTAC training data VALUE PROPOSITIONS Streamline JTAC/pilot comms. during CAS missions. Reduce pilots repeat- requests from the JTAC. Increase confidence in informational relay, correlation, and targeting: pilots, JTAC, civilians, all find this valuable. MISSION ACHIEVEMENT/IMPACT FACTORS Reducing cognitive load on Pilots, so they can better process ROE. JTACs require less time to talk pilots on to a target via radio. Yet, still retain digital efficiencies. KEY ACTIVITIES ● Implement ASR for pilot-JTAC comms ● Design simple UX with replay, aut- parsing, speaker notation, and customization. DEPLOYMENT Construct voice-to-text system running on pilot tablets. Link radio stream into tablet auxiliary input. On-device inference. BUY-IN & SUPPORT Work within key partners to integrate this product into the ATAK system. MISSION BUDGET/COST Maximize spending on data and engineers. Burn rate of $60,000/mo. BENEFICIARIES JTAC USAF Fighter Pilots Civilians KEY PARTNERS ● Air Force Research Laboratory ● 13th ASOS / 92nd AGOW ● JAIC ● 422nd Test & Evaluation Squadron
- 30. Finding Advocates, Building Prototype over Summer Seeking Funding Acquiring Data Building Partnerships
- 31. Looking Back Where we started: helping analysts save time Where we are now: helping pilots save civilian lives
- 32. Looking Back Where we started: helping analysts save time Where we are now: helping pilots save civilian lives Help super multi-taskers
- 33. Looking Back Where we started: helping analysts save time Where we are now: helping pilots save civilian lives Help super multi-taskers focus on the tasks that matter the most.
- 34. Huge Thank You to: Our Mentors: Rafi Holtzman, Jennifer Smith-Heys, Matthew ‘Fuego’ Tuzel Our Sponsors: Joe Murray, Matt Moore The Hacking for Defense Course Staff: Nate Simon, Sam Lisbonne, Valeria Rincon, Steve Blank, Steve Weinstein, Joe Felter, Pete Newell, Tom Bedecarré, Jeff Decker And Finally: Our classmates, and especially all of our interviewees.
Editor's Notes
- Hello, we are Team Protocol One, sponsored by the Joint Artificial Intelligence Center.
- Our overarching motivation was to protect civilian lives. Indeed, our name, Protocol One, is inspired by the Additional Protocol One to the Geneva Conventions, which prohibits harming civilians. Civilian Casualties, or CIVCAS is not only a moral and legal imperative to be concerned about. It is also a strategic priority. Civilian death leads to more people being driven to anti-U.S. forces, which in turn works against our objectives. It’s important to mention, however, that the vast majority of CIVCAS in conflict is not caused by US forces. The US is very attentive to protecting civilians. However, the US does make mistakes.
- For example, in addition to multiple smaller incidents, the two most notorious are: The accidental strikes of the Chinese Embassy in Belgrade in 1999 and of a Doctors without Borders in hospital in 2015.
- To understand why these accidents happen, we have interviewed 100 people, ranging from Intelligence Professionals to Pilots to experts from NGO, military, and Artificial Intelligence communities.
- This presentation will cover how we approached the problem of civilian casualties over the past 10 weeks.
- When we first entered the class, we were given a very specific problem statement: That geospatially defining No-Strike Facilities is time-consuming for analysts. This slide shows an example of a no-strike facility (specifically, a school), surrounded by a box defining its boundaries.
- Why are these boxes important? It turns out that knowing exactly how big the no-strike facility is helpful for targeting and weaponeering. Currently, analysts do this by hand, which is far too slow. The solution that was originally on our minds was to do this via an AI, which would turn an unbounded satellite image (as you see at left), to the bounded image at right. But we wanted to step back into the larger process of targeting, because our solution shouldn’t just be about saving analysts time. We wanted to know how this would impact civilian casualties.
- By talking to targeteers and looking at the lengthy Joint Targeting Manual, we pieced together this high level understanding of the Joint Targeting Cycle.
- We learned that the drawing of boundaries was a very small part in a very large process. Specifically, we gained two key insights.
- First, we learned that, throughout the process, intelligence supports decision making -- as you can see in these three highlighted areas.
- Where does this information come from? The Modernized Integrated Database. This is where the NSF Geospatial boundaries would be created, stored, and accessed before a strike. Information can come from various sources, as you can see on this slide.
- Our second key insight is that there is a difference between pre-planned strikes, which are typically done on fixed military targets that have gone through extensive analysis and vetting, and dynamic strikes, which are done on time-sensitive, potentially mobile, targets. But we found that relative to dynamic strikes, CIVCAS was very rare in pre-planned strikes.
- So although geospatially defining No Strike Facilities is an important part of the targeting process, we had difficulty connecting that task to downstream civilian harm. Analysts are able to draw the boundaries and that information is used. Instead, we shifted our focus to dynamic strikes, which we found to be the most relevant to CIVCAS because they are time-constrained and can occur in potentially unfamiliar locations, so they have a higher risk of mischaracterizing a target.
- For example, in war zones, buildings can often change functions due to the rapidly changing situations on the ground. A residential building may be used as a mosque if the mosque is under repair. Keeping up with these changes is a challenge.
- Or consider the example where you’re flying overhead at 2am and you see people digging at the side of the road. You might assume they’re planting an IED and are a threat, but if you were familiar with that village, you’d know that the locals dig their irrigation trenches at night since it’s too hot during the day. Our key takeaway from discussing these types of risks with pilots, troops, and targeteers was that understanding local civilian behavior can help prevent mistakenly targeting benign structures or activity.
- This idea was corroborated by a civilian protection specialist at the UN, who told us that “If you want to protect...life” This contrasts with traditional intelligence; although the MIDB has information on targets, it generally doesn’t describe the behavior of non-targets.
- This brings us to our second MVP, a crowdsourcing platform and database for localized and up-to-date information on civilian activity. Ground troops, local military partners, Open Source information (such as social media), and the civilians themselves would contribute to....
- ...a database of local population flows, cultural norms, and more...
- ...which is then vetted and refined by analysts...
- ...and made available to troops, pilots, and targeteers to provide potentially life saving civilian context for the missions they run. This represents a major divergence from the intelligence inside of MIDB, which does not generally tap these sources and can only be curated by professional intelligence analysts, which limits how fast it can be updated.
- Unfortunately, we found that people were lukewarm about this idea. They couldn’t point to concrete pain points in their workflows that our MVP would solve.
- In fact, most of the people we spoke with had trouble connecting their direct work to downstream civilian casualty incidents in general. Most were only familiar with a handful of incidents that were hard to generalize. So although we found a DoD report listing dozens of incidents with hundreds of casualties, our interviewees didn’t know the context for those incidents. And we were ultimately unable to connect with anyone who did know. We couldn’t take our theoretical MVP and map it to actual pain points that operators experienced, so our MVP development stagnated.
- ...and we pivoted once more.
- This MVP was the first where our end users –pilots and JTACs– were truly excited by what we built.
- The reason is that pilots no longer need to worry about perfectly copying everything down, even when the radio cuts out. And JTACs can now quickly send digital nine-lines, using only their voice.
- So now we’re looking to deploy our solution, with the help of partners across the air force and DoD.
- This summer we’re looking to raise money through AFSOC, AFWERX , and H4X Labs, which we’ll use to hire more engineers and process data. Our data has already, been provided by The 93rd Air Ground Operations Wing. A key to our success will also be working with the AFRL, who owns the tablet hardware, and the JAIC who can help us deploy our AI system. Finally, our beta testers will be the 13th air support ops squadron.
- Look, we learned a lot this quarter but looking back, what it’s boils down to is quite simple. We started by helping analysts save time. Which helped civilians downstream. Now we’re focusing on saving civilian lives directly, by aiding the pilot at the final link in the chain.
- The way we do this is by providing pilots with the data they need so they can focus on the tasks, like rules of engagement, that matter the most.
- The way we do this is by providing pilots with the data they need so they can focus on the tasks, like rules of engagement, that matter the most.
- We’re continuing this project, so if anyone in the audience can help us or knows someone with relevant experience, please email … Thank you!