ASRC Federal created the Mission Operator Assist (MOA) tool to extend human capabilities through AI/ML for NOAA. MOA ingests system log data from on-orbit satellite constellations and applies machine learning to greatly improve real-time situational awareness. MOA uses a collection of tools, including Kafka for multi-subscriber communications, all hosted through AWS Cloud Services and Kubernetes Containers for microservices. Like many traditional on-premises systems, satellite ground station operations are undergoing a renaissance as they increasingly become enabled by cloud. During this session, the audience will learn about the satellite communications chain, and best practices and lessons learned in creating a data pipeline with Kafka for high throughput and scalability while displaying high quality situational awareness to mission operators. We will discuss our goals centered around establishing event-driven streaming for satellite logs so our machine learning becomes real-time and supporting a multi-subscriber approach for various Kafka topics. Listeners will also learn how a multi-subscriber approach using Kafka, helped us auto scale logstash based on how many messages are in the queue and other microservices.

1. ASRC FEDERAL
Kafka and the Satellite Operations Domain
Artificial Intelligence, Machine Learning, and Streaming Support for
Satellite Data Operations
Eric Velte
Chief Technology Officer
ASRC Federal

2. Agenda 2
Overview of the Satellite Command and Data Acquisition (SCDAS)
domain
Opportunities for AI & ML and tools like Confluent and Kafka
Mission Operator Assistant Phase 1 – establishment of a Minimum
Viable Product (MVP)
What we learned from our initial prototype
Phase 2, deployment and future enhancements

3. ASRC Federal Proprietary
3
At a Glance
Our Federal Markets
• Space
• Defense
• Intelligence Community
• National Security
• Health
• Civilian
~8,000
Employees
44
States,
districts,
territories we
operate in
32
Subsidiary
companies
supporting
federal
government
2003
Founded
40%
Employees in
engineering,
development,
and analytics
46%
Employees
with Secret or
Top-Secret
clearances

4. Our Problem Space 4
More than 2,500 operating
satellites in space
Commanding, controlling,
and acquiring information is
a full-time job!
IT and Software have
evolved over time
ASRC Federal supports the engineering and day-to-day
operation of satellite constellations for NASA, NOAA, and others

5. Challenges – The Operations Landscape 5
• Traditional satellite data
ingestion, storage and
processing are very complex,
very human-intensive, somewhat
slow and very expensive
System undergoes
constant technical
refactoring and re-
imagining with modern
tools and techniques
NOAA Wallops CDAS,
Wallops Island, VA
Public Product Viewer –
GOES
Satellite Ground
Architecture – GOES-R

6. 101101101110011010100101
010010101001001001001111
The Legacy Implementation 6
Problem detection and troubleshooting
can take days to understand and solve
due to multiple challenges
<…
while (1)
grep “Error” goesr17/log.data>
end
…>

7. Improve Situational Awareness with MOA
Mission Operations Assistant
LACK OF VISIBILITY PUTS MISSIONS AT RISK
• Difficult for Operations to identify &
Analyze time-sensitive logfiles
• Lack of centralized log file structure,
arcane manual analytical procedures
• No easy way for the user to identify
when the message was seen last
GOAL OF MOA: EXTEND HUMAN CAPABILITIES THROUGH AI/ML
• Reduce the number of Tier-1 false positive error messages or indicators
• Minimize escalation to Tier-2 (OSPO/EMOSS), Tier-3 (Maint. & Sust.) and Tier-4 (factory)
• Leverage ML to minimize log-file noise, improve troubleshooting efficiently & identify impact across
assets
Loss of
situational
awareness
Time
consuming
manual
effort
Log file
overload
Missing
Critical
Alerts

8. Artificial Intelligence in the Satellite Domain
Bringing Efficiency to Satellite
Mission Operations
8
NOAA’s Satellite Operations
Facility is a staffing-
intensive operation
Satellite mission challenges
are hard to detect,
troubleshoot and repair
Introduction of automation
reduces troubleshooting
time from days to hours.
ASRC Federal introduced automation to assist
operator in detecting operations and control issues
Pixel Striping Anomaly

9. Human in the Loop –
Operator Alerting
9
Reaction wheels wear out with use.
Team was able to determine reaction
wheel slippage and develop an
operating procedure that minimized
wear-and-tear, potentially
prolonging spacecraft life
AI & ML are also useful tools for the
prediction of failures

10. The MOA Minimum Viable Product (MVP) 10
Log & TT&C Data
MOA Front End
Satellite-Specific Insights

11. MOA Features
SEARCH ENGINE
ALARM VIEW
IMPACT MAP
• Enhanced search capabilities
• Build centralized knowledge base by allowing
user to add notes to event type
• Show graphical timeline of occurrences for
improved visualizations
• Instrumented to capture user interaction for ML training
• Build centralized knowledge base by allowing user
to add notes to event type
• Show graphical timeline of occurrences for improved
visualizations
• ML enabled view to minimize extraneous noise on display
• Leverage ML to predict impacts prior to
completion of product generation
• Maintains a history of impacts for easy
reference
Prototype screenshots

12. Tools in our Development and Operations Space
Our AI/ML programs
Cloud native AI / ML solutions such as AWS
SageMaker, MS Cognitive Services, and Google
Analytics are all big enablers of mission
success. We also develop our own domain-
specific & custom AI/ML frameworks
Continuous Integration and Continuous
Development help us design, deliver,
deploy, monitor and maintain at the speed
of need!
*Bounded Computational AI & Cognitive Computing

13. Satellite Data Log Management
From Ingest to Insight – Phase 1 (Completed 2Q2021)
13
1. Ingest of log information from
server-based log files
1
2
3
4
2. Correlation of data across
multiple sources using Kafka
3. Storage of data into Elastic and
Postgres containers
4. Asynchronous access and
update of data from the front
end
• Enables us to use event driven streaming of satellite logs
• Enables us to easily scale Logstash, backend, and other microservice instances
• Redundancy, stability, fault-tolerant for client services
• Allows multiple consumers for each topic

14. Satellite Data Log Management
Microservices Communications and Segmentation – Phase 2
14
• Driving towards a real-time system for
anomaly detection and problem resolution
• Current tools are tightly coupled and rely
on proprietary communications middleware
like TCP and direct connections between
components
Using Kafka:
• Promotes information sharing to broader
communities
• No need to determine who they are (policy)
or where the users are located
• Scalability improvements for message
surges
Phase 2: Expand Kafka into the microservices
environment as a total system messaging bus

15. Satellite Data Pipeline
Partnered and working with Confluent to expand
Kafka into the microservices environment
Evaluating Confluent
components:
• Confluent for Kubernetes for scalable, elastic
deployment and management
• RBAC for role management and access to
specific topics
• Kafka Connect to quick and reliable
connectivity to external data sources
• Schema Registry and Schema Validation for
satellite model management and evolution
• Kafka Streams and ksqlDB for real-time data
enrichment, aggregation, and filtering
• ksqlDB UDFs for anomaly detection
algorithms

16. MOA SOLUTION
• Automatic real-time aggregation of multi-
system log files
• Enhanced search capabilities
• Correlate impact analysis
• Identify historic patterns & predict analytics to
avoid future mission critical outage
• Extend human capabilities through the
use of AI/ML
VALUE TO CUSTOMER
• Risk mitigation and cost effectiveness.
• Extend life of critical satellite constellations
• Increased situational awareness satellite
h&s
• Consistent results across user base
• Proactive preemptive maintenance
• Centralized source of best practice
knowledge base
• Designed to be used for any mission w/
standardized logging systems
16
MOA satisfies unmet needs identified at 2017 GSAW :
“We think that the set of events/log message enhancements will provide powerful capabilities
for the mission user regardless of Agency or type of mission.”
“Placing an emphasis on non-telemetry analysis opens up a new area of data mining, analytics
and tool development [in satellite operations]–we think the users will help identify even more
functions.”
https://ntrs.nasa.gov/citations/20170002304

17. NOAA’s End-State Vision
Common Cloud Framework
17
Many types of satellite data are a
public commodity
• Weather
• Oceanographic
• Imagery
Traditional data ingestion, storage
and processing are cost-effective and
performed as-a-service
Cloud-based solutions enhance data
proximity & enhance user experience
Kafka is a key to the framework of
the future
Security and role-based access are
keys
Streaming solutions like Kafka and
Confluent tooling simplify and speed up
data flows from ingest to dissemination

18. Moving Forward with
ASRC Federal
18
Alaskan company with more than
8,000 employees
• Engineering
• Software
• Digital Modernization
• Professional Services
• Infrastructure
More than 1,500 employees working
with NASA, NOAA and others
20+ year history solving complex
technology and software problems
for the federal government

19. Summary 19
ASRC Federal is partnered with NASA and NOAA to modernize the
ground station architecture in accordance with agency goals and
roadmaps
Cloud migration and modern tooling are both keys to that success,
including better dependability and an anticipated cost reduction
We successfully used Kafka to implement the MOA message-passing
infrastructure in a phase 1 MVP; seeking to expand its use in follow-
on development

20. Quyanaq!
Thank you!
Eric Velte
Chief Technology Officer
e: evelte@asrcfederal.com