This document discusses using ADS-B data from a home plane spotting setup to send alerts when a specific plane's call sign is detected flying overhead. It considers various technologies like the Tick Stack and Kafka for ingesting and analyzing the streaming ADS-B data in real-time. The author tests ingesting the data into InfluxDB on AWS and viewing it in Grafana Cloud but does not get alerts working in the limited time. Lessons are learned around debugging time series data and challenges with free cloud tiers.

1. Plane Spotting
Hacking The ADS-B Stream
Columbus Ohio
Columbus Apache Kafka Meetup
MODUG: Mid-Ohio Data User Group
October 15, 2019
Ted Coyle
Solution Architect
Navigator Management Partners, now Avaap
How to Get a Free Enterprise
FlightAware Account

2. 1
Agenda
Tech Choices
Event Driven Alerts
Results
ADS-B Story
The Hack
Hacking the Stream
4 min
6 min
4 min
3 min
2 min

3. 2
Overview
Automatic dependent surveillance—broadcast (ADS–B)
ADS What?
ADS–B is an element of the United States Next Generation
Air Transportation System (NextGen)
• Automatic: Twice per second transmissions no operator input required
• Dependent: Position and velocity vector are GPS derived.
• Surveillance: Real-time aircraft positions
• Broadcast: Open transmission, anyone can receive the signals
ADS-B Radio
Receivers (IoT)
GPS
1090 MHz
Broadcast
Ground stations relay
data to FAA

4. 3
Physical Hardware
$30 $20 $0-$20
$1-$6
Pi 2 or 3 ADS-B 1090 Receiver
Hardware Costs should be under $70
ADS-B Receiver Hardware

5. 4
Hardware and Software
Hardware Setup
Total Setup Typically Under
$60-$70
Software is Free
Dump1090-fa
Choose either PiAware Image or Handspun Dump1090
All Inclusive PiAware Image

6. 5
User StoryWe want to know if my dog
really does sense me flying
over the house.
Birch is here
I’m about a mile away
going 340 mph at 8k ft.

7. 6
User StoryWe want to know if my dog
really does sense me flying
over the house.
Birch is here
I would like to use my
FlightWare 1090dump-fa
stream to alert when my
plane is overhead.

8. 7
Story Points
Google Earth
8000 ft View
Constant:
Approx. 40-80 minutes travel
post fly-by depending on
landing delays, traffic, etc.
Repeated Behavior:
Birch waiting by door after
plane flew over (9:21 am).
Variable:
Almost always on
different flights so no way
Birch can correlate by
Time of Day arrival.

9. 8
Tech Choices
• Already have PiAware Running
• Two stream types – 1090dump
• Port 30005 - Binary Beast Format
• Port 30003 CSV
• Don’t want to break existing PiAware Setup
• Don’t want audible or other annoying alert
• Have access to every cloud… GCP, AWS, OCI, Azure
• Have a bunch of friends in tech…
• This is a Kafka/data meetup…
• Really want to dive into the Tick Stack
Considerations

10. 9
Tech Choices
• What Cloud-Native Services Could I use?
• Tick Stack… knew parts, but not other parts.
• New PC… Setup issues, accounts, config
• Could I get Telagraf working to Confluent Cloud?
• Could I get a new test PiAware going?
Considerations – Things I didn’t know

11. 10
Tech Choices
Easy Path – The ASCII Text Stream
5 Seconds of Dump1090-fa port 30003 output

12. 11
Tech Choices
Easy Path – The Stream Elements
5 Seconds of Dump1090-fa port 30003 output

13. 12
Tech Choices
Easy Path – The Stream Elements
• I only really care about the call sign.
• If it appears, it means my plane is
within 50 miles of my house.
• Fire alert if my plane’s call sign is
picked up by my local receiver.

14. 13
Tech Thoughts
FlightAware Range and Volume
Poor Antenna Placement

15. 14
Tech Thoughts
As the Crow Flies
Typical Flight Paths

16. 15
Tech Thoughts
As the Crow Flies

17. 16
Tech Choices
Keep It Simple – Initial Thoughts
Port 30003
CSV
Python
Scripts
Ingest
Pipeline
Data Flow and Time
Series
Alert

18. 17
Tech Parts
TimeSeries – The Tick Stack
Open Source
plugin-based
data collector
World Class
Open Source
Time Series
Database
Alerting

19. 18
Tech Parts
TimeSeries – Dashboards

20. 19
Tech Choices
What is the schema?
Which tool for ingest?
How to alert?
Considerations

21. 20
Tech Reality
Considerations - Slightly Less Simple
Port 30003
CSV
Python
Scripts
Ingest
Pipeline
Data Flow and Time
Series
Python
Telegraph

22. 21
Tech Choices
DuckDuckGo to the Rescue
Search revealed two already built
ingest methods…
1. Direct
2. Telegraf
https://duckduckgo.com/?q=adsb+influx&t=hk&ia=software

23. 22
Tech Choices
Considerations
https://github.com/mikenye

24. 23
Tech Choices
Considerations – Ingest
Python
Scripts
Schema Parsing
Code
Python
Telegraf
So that
settles it, I’ll
use Python
and Telegraf
to insert into
InfluxDB.

25. 24
Tech Choices
Considerations – Stream Ingest
Python
Scripts
Ingest Tool

26. 25
Tech Choices
Telegraf Config File
TAML File
Ingest Tool
TAML is an acronym of Torque Application Mark-
up Language. It decribes a method of persisting
type instances in various output formats. TAML is
designed to be extremely easy to use and
provides a nice consistent method of persisting
information.

27. 26
Tech Choices
Considerations – File Based Ingest
Python
Scripts
Options…Batch versus Stream
Lots of ways to do this
JSON Files

28. 27
Time Series... Running out of Time
• Fix Dump Stream…Linksys Router Bug.
• Get Influx Running..
• Docker Annoyances
• Alter Alert Script
• Can Telegraf Be the Alerter?
• Where to Stream?
• Cloud?
Demo Compromise

29. 28
Time Series... Running out of Time
• Fix Dump Stream…Linksys Router Bug.
• Get Influx Running..
• Docker Annoyances
• Alter Alert Script
• Can Telegraf Be the Alerter?
• Where to Stream?
• Cloud?
Demo Compromise
• No… Use Kapacitor
Influx,Kafka on AWS
Grafana Cloud

30. 29
ADS-B Easy Ingest
docker run ^
-d ^
--name=piaware2influx ^
--restart=unless-stopped ^
-e INFLUXDB_URL="http://ec2-54-89-244-105.compute-1.amazonaws.com:8086" ^
-e DUMP1090_HOST="skyminate.com" ^
-e TZ="America/New_York" ^
mikenye/piaware-to-influx
Docker-PiAware-to-Influx In Action

31. 30
ADS-B Ingest
docker run ^
-d ^
--name=piaware2influx ^
--restart=unless-stopped ^
-e INFLUXDB_URL="http://ec2-54-89-244-105.compute-
1.amazonaws.com:8086" ^
-e DUMP1090_HOST="skyminate.com" ^
-e TZ="America/New_York" ^
mikenye/piaware-to-influx
Docker Image - Out of the Box Easy

32. 31
ADS-B Ingest
docker run ^
-d ^
--name=piaware2influx ^
--restart=unless-stopped ^
-e INFLUXDB_URL="http://ec2-54-89-244-105.compute-
1.amazonaws.com:8086" ^
-e DUMP1090_HOST="skyminate.com" ^
-e TZ="America/New_York" ^
mikenye/piaware-to-influx
Docker Image - Out of the Box Easy

33. 32
ADS-B Ingest
Home Router – Port Forwarding
http://skyminate.com/

34. 33
ADS-B Ingest
Influx Database – PiAware Schema
name: piaware
fieldKey fieldType
-------- ---------
current_altitude float
current_groundspeed float
current_latitude float
current_longitude float
current_track float
current_verticalrate float
> show tag keys
name: piaware
tagKey
------
callsign
hexident
squawk
>
Indexed
Not
Indexed

35. 34
Lessons Learned
• Nothing’s Easy, except when it is.
• Spent a lot of time looking at the data
• Could have spent more time on Ops
• New Laptop slowed pace
• Automation is key to everything
• Except when debugging

36. 35
Lessons Learned
• The Tick Stack
• Grass-roots command line
• Debugging is not straight forward
• Stick to the happy path
• Same OS, Base Tests, etc.
• I’m not good with TimeSeries Queries yet

37. 36
Lessons Learned
• Free Cloud Tiers
• Maybe Rethink that…
• It’s free for a reason.
• Very easy to mess Grafana up and get stuck or lose
work.

38. 37
Lessons Learned
I did not get alerts
working....yet….
Example of not
following the happy
path ---------

39. 38
Lessons Learned
Kapacitor Streaming
Alerts.
Example of not
following the happy
path ---------

40. 39
Lessons Learned
Grafana and Influx Are Meant for Each Other

41. 40
Lessons Learned
Debugging in Chronograf Looks Promising

42. 41
Hacked Architecture
Demo Hack Physical Architecture
This is where it all landed…
Yum installed InfluxDB running on t2.large instance
InfluxDB
Grafana Cloud
Influx tools also installed locally to execute influx
queries from command line
PiAware
Skyminate.com
Time Spent… approx. 6 hours

43. 42
Onward
• Github https://github.com/EscVector/ads-b
• Terraform/Ansible Instance Spinup
• Extract/merge good parts from existing projects
• K8s for Kafka/Confluent/Influx
• Telegraf ADS-B Input Plugin

44. • Blog - ADS-B Deep Dive
• Decoding Mode-S and ADS-B
• FlightAware ProStick
• FlightAware Building a PiAware Receiver
• Influx and ADS-B
• Influx Telegraph to Kafka and InfluxDB
• Influx Industry 4.0 Overview
• PlanePlotter for Windows (25 Days Free)
• 1090 MHz Antenna Building – Video Example
• Kafka ADS-B
• LiveATC – Air Traffic Control
43
Appendix A
Resources

45. • Telegraf Kafka Plugin
• https://github.com/slintak/adsb2influx
• Wikipedia ADS-B
• Next Generation Air Transportation System
• https://github.com/just-containers/s6-overlay
• https://www.influxdata.com/blog/tldr-influxdb-tech-tips-august-4-
2016/
• https://docs.influxdata.com/influxdb/v1.7/concepts/key_concepts
/#field-key
44
Appendix B
Resources

46. 45
Appendix C
Coax Linear Antenna

47. 46
Appendix D
Telegraf Setup – Confluent Cloud not Working
Docker cp telegraf.conf /etc/telegraf/telegraf.conf
[[outputs.influxdb]]
urls = ["http://ec2-54-89-244-105.compute-1.amazonaws.com:8086"]
database = "piaware"
skip_database_creation = false
timeout = "5s"
# Influx HTTP write listener
[[inputs.http_listener]]
service_address = ":8186"
read_timeout = "10s"
write_timeout = "10s"
#[outputs.kafka]
# URLs of kafka brokers
# brokers = ["pkc-438qm.us-east-1.aws.confluent.cloud:9092"] # EDIT THIS LINE
# Kafka topic for producer messages
# topic = "piaware"
# #ssl_endpoint_identification_algorithm = "https"
# sasl_mechanism = "PLAIN"
# request_timeout_ms = 20000
# # bootstrap.servers=pkc-438qm.us-east-1.aws.confluent.cloud:9092
# retry_backoff_ms = 500
# sasl_jaas_config = "org.apache.kafka.common.security.plain.PlainLoginModule required username="key" password="keypass";"
# security_protocol = "SASL_SSL"

48. 47
Appendix - Tech Thoughts
FlightAware versus ADSB-Exchange
Contributing to ADS-B Exchange would be cool,
but maybe next week...