Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
High-Throughput Processing of
Space Debris Data
Andreas Schreiber, Michael Meinel
German Aerospace Center (DLR)
Simulation...
• Space Debris
• BACARDI
• Skynet
Outline
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Dat...
Major research areas of DLR
• Aeronautics
• Space
• Transportation
• Energy
• Security
Software research and dev.
• Softwa...
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 4
Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 5
Lower Orbit...
Space Flight
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 6
• Almost 5,000 rockets launched
• More than 6,000 satellites placed in orbit
• About 1,000 active satellites today
(that’s...
Defunct objects in space
• Disused satellites & rocket stages
• Fragments from explosions &
collisions
• Released object (...
Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart ...
Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart ...
16,300 Objects in Catalogue
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015...
29,000 Objects Larger than 10 cm
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07...
750,000 Objects Larger than 1 cm
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07...
150M Objects Larger than 1 mm
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.20...
Impact
Laboratory Impact Aluminum Sphere / Block
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Deb...
Impact
Real Debris Object on a Satellite Solar Panel
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space...
Distribution of Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.201...
Collision Prediction
International Space Station
DLR.de • Chart 18 > PyData Seattle > A. Schreiber • High-Throughput Proce...
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 19
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 20
http://rbt...
Detecting Space Debris
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.d...
SMARTnet
• Theory: complete coverage
with 3 locations
• Northern / southern hemisphere
for compensating seasonal
variation...
Operational Collision Avoidance
Global network for monitoring the geostationary ring
> PyData Seattle > A. Schreiber • Hig...
Object Identification with Optical Measurements
DLR.de • Chart 24 > PyData Seattle > A. Schreiber • High-Throughput Proces...
• Separation of real / false tracklets above threshold of loss
function (chi-squared distribution)
• Filter rate depends o...
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 26
BACARDI
Objective
• Database with preferably high completeness and high
accuracy
• Primary source: sensor data and operator data
•...
Science and Research
• Database of more than 1M objects
• High-performance computational…
• object correlation
• orbit det...
Mission Support
• Orbit information
• Collision prediction
• Re-entry prediction
Real-time operation
BACARDI
Backbone Cata...
BACARDI Overview
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Ch...
BACARDI Overview
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Ch...
BACARDI
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 32
External Data Sources
Example: space-track.org
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debri...
BACARDI
System Components
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DL...
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 35
Skynet
Specialized middleware with following objectives
• Decentralized message queues
• Scalable, self-organizing network
• Mini...
ZeroMQ
• Decentralized network infrastructure and messaging
Protocol Buffers
• Highly efficient serialization of data
SQLA...
Skynet
Modules
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Char...
• Socket library for messaging
• Multi-platform, multi-language
• Fast and small
• Many connection patterns
ZeroMQ
www.zer...
• Network layer based on ZeroMQ
• „Device“ as abstract endpoint
• XML configuration: ZDCF (ZeroMQ Device Configuration Fil...
skynet.network
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Char...
• Specified by Google (https://github.com/google/protobuf)
• Serializing (encoding) structured data
• Efficient and extens...
• All models are defined as Protocol Buffer messages
• Provides basic data type as ProtoBuf (e.g., 3D vector, 6D
vector, o...
If a model should be mapped to a database table, it should
have the db_table message option extension set
skynet.model
Mod...
skynet.model
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart ...
// Orbits and orbital parameters.
package bacardi.model.orbit;
import "skynet/model/proto/options.proto";
import "bacardi/...
skynet.model
Initialize a Database with All the Models
> PyData Seattle > A. Schreiber • High-Throughput Processing of Spa...
Two ways of accessing data
• Sending a Query to the DataStore
• Using the db extension
• More compact code and hence bette...
Abstraction layer for processes
• Decoupling of processes from network layer
• Automatic caching, if needed
Easy integrati...
Provenance is defined as a
record that describes
the people, institutions,
entities, and activities
involved in producing,...
Provenance recording
• Recording of all activities during runtime
• Multiple Provenance stores
• Provenance information st...
Technical
• Graphical User Interfaces
• Web (Django) and desktop (PyQt)
• Workflow
Organizational
• Operational for simple...
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 54
Thank You!...
Upcoming SlideShare
Loading in …5
×

High Throughput Processing of Space Debris Data

1,261 views

Published on

Space Debris are defunct objects in space, including old space vehicles or fragments from collisions. Space debris can cause great damage to functional space ships and satellites. Thus detection of space debris and prediction of their orbital paths are essential. The talk shows a Python based infrastructure for storing space debris data from sensors and high-throughput processing of that data.

PyData Seattle (26. Juli 2015)
http://seattle.pydata.org/schedule/presentation/35/

Published in: Software
  • Be the first to comment

High Throughput Processing of Space Debris Data

  1. 1. High-Throughput Processing of Space Debris Data Andreas Schreiber, Michael Meinel German Aerospace Center (DLR) Simulation and Software Technology, Berlin / Cologne, Germany PyData Seattle July 26, 2015 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 1
  2. 2. • Space Debris • BACARDI • Skynet Outline > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 2
  3. 3. Major research areas of DLR • Aeronautics • Space • Transportation • Energy • Security Software research and dev. • Software engineering, HPC, distributed systems, … DLR Aerospace Research Center, Space Agency > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 3
  4. 4. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 4 Space Debris
  5. 5. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 5 Lower Orbit 300 – 2000 km Geostationary Orbit ≈36,000 km
  6. 6. Space Flight > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 6
  7. 7. • Almost 5,000 rockets launched • More than 6,000 satellites placed in orbit • About 1,000 active satellites today (that’s 7% of known objects in orbit) About 50 Years of Space Flight > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 7
  8. 8. Defunct objects in space • Disused satellites & rocket stages • Fragments from explosions & collisions • Released object (tools etc.) • Slag of solid-fueled rockets • … and other sources of small particles Space Debris > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 8
  9. 9. Space Debris > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 9 January 2007: FengYun-1C February 2009: Collision Iridium-33 & Kosmos-2251
  10. 10. Space Debris > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 10 Iridium-33 & Kosmos-2251: Debris fields after 50 minutes Fengyun-1C debris one month after its disintegration
  11. 11. 16,300 Objects in Catalogue > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 11
  12. 12. 29,000 Objects Larger than 10 cm > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 12
  13. 13. 750,000 Objects Larger than 1 cm > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 13
  14. 14. 150M Objects Larger than 1 mm > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 14
  15. 15. Impact Laboratory Impact Aluminum Sphere / Block > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 15
  16. 16. Impact Real Debris Object on a Satellite Solar Panel > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 16
  17. 17. Distribution of Space Debris > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 17 Source: NASA Orbital Debris Quarterly News 2/2012 ISS Many Earth Observation Satellites (e.g., Envisat) TerraSAR-X
  18. 18. Collision Prediction International Space Station DLR.de • Chart 18 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015
  19. 19. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 19
  20. 20. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 20 http://rbth.com/news/2015/07/26/iss_successfully_ducks_from_space_debris_48023.html
  21. 21. Detecting Space Debris > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 21 Sources: FhG FHR, AIUB, ESA Optical Radar Laser
  22. 22. SMARTnet • Theory: complete coverage with 3 locations • Northern / southern hemisphere for compensating seasonal variations (6 locations) • Telerobotical operation • Optimized scheduler for all telescopes Operational Collision Avoidance Global network for monitoring the geostationary ring > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 22
  23. 23. Operational Collision Avoidance Global network for monitoring the geostationary ring > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 23
  24. 24. Object Identification with Optical Measurements DLR.de • Chart 24 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015
  25. 25. • Separation of real / false tracklets above threshold of loss function (chi-squared distribution) • Filter rate depends on accuracy, time difference, survey strategy, … Object Correlation > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 25
  26. 26. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 26 BACARDI
  27. 27. Objective • Database with preferably high completeness and high accuracy • Primary source: sensor data and operator data • Secondary source: externally generated ephemerides BACARDI Backbone Catalogue of Relational Debris Information > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 27
  28. 28. Science and Research • Database of more than 1M objects • High-performance computational… • object correlation • orbit determination • propagation • object identification • detection of maneuvers and fragmentations BACARDI Backbone Catalogue of Relational Debris Information > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 28
  29. 29. Mission Support • Orbit information • Collision prediction • Re-entry prediction Real-time operation BACARDI Backbone Catalogue of Relational Debris Information > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 29
  30. 30. BACARDI Overview > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 30
  31. 31. BACARDI Overview > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 31 Sensor network BACARDI Sensor scheduler User Groups External orbital / object information Optical Radar Laser Database User Interface Processes
  32. 32. BACARDI > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 32
  33. 33. External Data Sources Example: space-track.org > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 33
  34. 34. BACARDI System Components > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 34 RDBMS Cache Import Processing Python FORTRAN Middleware Security Traceability • Simple • Parallel • Fast • Huge number Export
  35. 35. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 35 Skynet
  36. 36. Specialized middleware with following objectives • Decentralized message queues • Scalable, self-organizing network • Minimal network overhead • Platform independent • Operation: Linux • Development: Windows, Mac OSX • Automatic provenance recording Skynet Network for surveillance of the sky > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 36
  37. 37. ZeroMQ • Decentralized network infrastructure and messaging Protocol Buffers • Highly efficient serialization of data SQLAlchemy • Database mapping ZeroMQ and Protocol Buffers are available for a wide range of platforms and programming languages. Skynet Technology Decisions > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 37
  38. 38. Skynet Modules > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 38 skynet skynet.network Messaging skynet.model Data Model skynet.process Processes skynet.record Provenance
  39. 39. • Socket library for messaging • Multi-platform, multi-language • Fast and small • Many connection patterns ZeroMQ www.zeromq.org > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 39
  40. 40. • Network layer based on ZeroMQ • „Device“ as abstract endpoint • XML configuration: ZDCF (ZeroMQ Device Configuration File) • Automatically connected to other matching „devices“ (Peer-to-Peer) • Compression of large data packets • (De-)serialization via Protocol Buffer messages • Prepared for encryption and signing • Synchronous or asynchronous skynet.network > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 40
  41. 41. skynet.network > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 41
  42. 42. • Specified by Google (https://github.com/google/protobuf) • Serializing (encoding) structured data • Efficient and extensible • Language-neutral, platform-neutral • Interface definition language to describe data structure Protocol Buffers (“ProtoBuf”) > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 42 message Person { required string name = 1; required int32 id = 2; optional string email = 3; }
  43. 43. • All models are defined as Protocol Buffer messages • Provides basic data type as ProtoBuf (e.g., 3D vector, 6D vector, orbital data, …) • Interfaces for caching • ORM for ProtoBuf messages with SQLAlchemy (database- neutral) • Very few additional annotation • Compiled to Python module using protoc • skynet.network device for database connection skynet.model > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 43
  44. 44. If a model should be mapped to a database table, it should have the db_table message option extension set skynet.model Model Definition > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 44 // Commonly used message types. package bacardi.model.common; import "skynet/model/proto/options.proto"; // A spatial vector with x, y and z double components message vec3d { required double x = 1; required double y = 2; required double z = 3; }
  45. 45. skynet.model > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 45 BACARDI Network Message HEADER BLOCK 1 BLOCK 2 BLOCK N NULL… DEST NULL SOURCE NULL… … TYPE COMMAND NULL [SIG]…[KEY]
  46. 46. // Orbits and orbital parameters. package bacardi.model.orbit; import "skynet/model/proto/options.proto"; import "bacardi/model/common.proto"; // One entry of a propagation error dataset. message properr_item { required double days_prop = 1; required double altitude = 2; required double mean_f107 = 3; required common.vec6d std_state_rtn = 4; } // Propagation error dataset. message properr { option (skynet.model.db_table) = { name: "properr"; }; Model Example Orbits and Orbital Parameters > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 46
  47. 47. skynet.model Initialize a Database with All the Models > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 47 # Import skynet ORM and database. from skynet.model import mapping, db # Search for models in 'bacardi.model'. # Alternatively, you could also pass a Python module. mapping.Mapper.scan_module('bacardi.model’) # Connect to SQLite database store = db.DataStore('sqlite:///bacardi.db') # Create all tables (SQL schemas) for # the registered models. store.create_tables()
  48. 48. Two ways of accessing data • Sending a Query to the DataStore • Using the db extension • More compact code and hence better readability • To retrieve the ISS data, one could simply: skynet.model Data access > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 48 # Import data model for objects. from bacardi.model import object_pb2 # Get the obj with NORAD ID 25544 (ISS Zarya). iss = object_pb2.obj.get(norad_id=25544)
  49. 49. Abstraction layer for processes • Decoupling of processes from network layer • Automatic caching, if needed Easy integration of processes • as sub process (data via disk I/O or named pipes) • as FORTRAN or C module with Python wrapper • as pure Python module Worker device for multiple processes • One device per CPU / Core skynet.process > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 50
  50. 50. Provenance is defined as a record that describes the people, institutions, entities, and activities involved in producing, influencing, or delivering a piece of data or a thing. (W3C Provenance Working Group, http://www.w3.org/2011/prov) Provenance > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 51 Agent Entity Activity used wasGeneratedBy wasDerivedFrom wasStartedBy wasEndedBy wasAssociatedWith actedOnBehalfOf
  51. 51. Provenance recording • Recording of all activities during runtime • Multiple Provenance stores • Provenance information stored as graph in Neo4j Traceability and provability of all processed data • Backtracking of each produced product (ephemerides, state vectors, correlated objects, …) • Reproducibility of products and data generated Work-in-progress skynet.record > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 52
  52. 52. Technical • Graphical User Interfaces • Web (Django) and desktop (PyQt) • Workflow Organizational • Operational for simple workflows end of 2015 at DLR • Extended SSA versions for DLR and German Army • Other use cases for Skynet (e.g., data acquisition for rocket motors) BACARDI and Skynet Current and Future Work > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 53
  53. 53. > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 54 Thank You! Questions? Andreas.Schreiber@dlr.de www.DLR.de/sc | @onyame

×