Data Con LA 2020 Description Amid the recent COVID-19 pandemic, we are curious to explore and visualize its impact on global airline traffic such as flight frequencies, volume and schedules. We are also interested in utilizing non-flight-related datasets in this analysis from organizations such as the John Hopkins University Center for Systems Science. OnPrem Solution Partners has built a fully operationalized data ingestion, transformation, and analysis platform using modern tools such as Snowflake, dbt, and Streamsets Data Collector to capture and analyze flight data. Our primary data source consists of minute-by-minute flight data from flights globally. We have been collecting and storing this state data since November 2018 and now have billions of records accumulated for analysis. In addition, we load auxiliary data into Snowflake to form a comprehensive analysis and reporting platform. We have explored the following use cases: *How the overall number of flights changes as the virus spreads? *How do the number of flights change for a specific region as the number of positive tests/deaths change? *How does the number of flights to/from countries with the highest number of cases (hot zones) change? Speaker Yasha Mouradi, OnPrem Solution Partners, Data & Analytics Manager

1. onprem.com
1

2. onprem.com
2
●
●
●
●
●
○

3. onprem.com
3

4. onprem.com
4
1

5. onprem.com
5
1
2

6. onprem.com
6
Other positive developments presented in the outlook
include increased connectivity, with airlines
connecting more cities than ever and at a lower
cost. IATA forecasts that this will rise above 23,000 for
the first time in 2020.
Emerging markets are experiencing the strongest
growth in air travel, with China leading the way at
8.5% year-on-year from October 2018 to 2019.

7. onprem.com
7

8. onprem.com
8

9. onprem.com
9

10. onprem.com
10

11. onprem.com
11
SHOW ME THE STACK!

12. onprem.com
12
Technical
○ Scalability of data ingest and processing from 10s to 100s of billions of rows
○ Maintaining known good state of system - version control or otherwise
○ Automated testing of data
○ Economical processing/run costs
○ Possibility of near-real time ingest if needed in the future
○ Avoid development of platforms and tooling where possible
○ Modular in stack where possible
Staffing/Organization
○ Focus on SQL - easier to staff and easier to train
○ User expectation of rapid changes/updates - days/weeks not months/years

13. onprem.com
13

14. onprem.com
14
- ingest data as raw as possible
- only perform required data type changes

15. onprem.com
15
- scalable, flexible technologies
- minimal infrastructure upkeep
- abstracted storage and compute
- ingest data as raw as possible
- only perform required data type changes

16. onprem.com
16
- scalable, flexible technologies
- minimal infrastructure upkeep
- abstracted storage and compute
- ingest data as raw as possible
- only perform required data type changes
- modern software development best-practices
- version controlled data processing scripts
- data pipelines testable and integrated into CI/CD frameworks
- data outputs monitored for data quality and errors

17. onprem.com
17
- modern software development best-practices
- version controlled data processing scripts
- data pipelines testable and integrated into CI/CD frameworks
- data outputs monitored for data quality and errors
- scalable, flexible technologies
- minimal infrastructure upkeep
- abstracted storage and compute
- ingest data as raw as possible
- only perform required data type changes
- tech stack usable by a variety of consumers
- data models are easy to join and query
- collapse the division between
"thinkers/planners" and "doers"

18. onprem.com
18

19. onprem.com
19

20. onprem.com
20

21. onprem.com
21

22. onprem.com
22

23. onprem.com
23

24. onprem.com
24
opensky-network.org

25. onprem.com
25
● Filter/cleanse/deduplicate raw OpenSky data (~50% reduction)
● Group sets of flight points down into a set of flight legs
● For each flight leg, locate the start/end airports (1 degree box)
● Group sets of flight legs into flights (flights with layovers)
A
B C
1x1 box
B
A
Airport
Flight Point
Flight Leg
Flight

26. onprem.com
26

27. onprem.com
27

28. onprem.com
28

29. onprem.com
29
1
2
3
- ingest data as raw as possible
- only perform required data type changes

30. onprem.com
30

31. onprem.com
31
dbt is an open source tool for authoring, testing, and orchestrating data engineering pipelines.
Key strengths:
- Data transformations are written using modular, performant, maintainable SQL
- Natively integrates with version control
- Integrated unit testing framework
- Uses the Jinja templating language to take advantage of sophisticated macroing capabilities
- Continuous Integration / Continuous Deployment (CI/CD)
- Metadata catalog / automated documentation
- Native logging / alerting capabilities
- dbt Core - open source / no licensing costs

32. onprem.com
32

33. onprem.com
33

34. onprem.com
34

35. onprem.com
35

36. onprem.com
36

37. onprem.com
37

38. onprem.com
38

39. onprem.com
39
Disconnected Tables
● Constants
● Parameters
● Measures
Groupings

40. onprem.com
40
Disconnected Tables
● Constants
● Parameters
● Measures
Groupings
Dimensions
Facts

41. onprem.com
41
SHOW ME THE DATA!

42. onprem.com
42

43. onprem.com
43

44. onprem.com
44

45. onprem.com
45

46. onprem.com
46

47. onprem.com
47

48. onprem.com
48

49. onprem.com
49

50. onprem.com
50

51. onprem.com
51

52. onprem.com
52

53. onprem.com
53

54. onprem.com
54

55. onprem.com
55

56. onprem.com
56

57. onprem.com
57

58. onprem.com
58

59. onprem.com
59

60. onprem.com
60

61. onprem.com
61

62. onprem.com
62

63. onprem.com
63

64. onprem.com
64
cruise and step climb(s)
optimum altitude (baseline)
actual altitude (baseline)

65. onprem.com
65
cruise and step climb(s)
optimum altitude (baseline)
actual altitude (baseline)
optimum altitude (pandemic)
actual altitude (pandemic)

66. onprem.com
66

67. onprem.com
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
67

68. onprem.com
68
Richard Branson
Founder, Virgin Atlantic

69. onprem.com
69

70. onprem.com
70

71. onprem.com
71

72. onprem.com
72
Dimension
Fact
Dimension
Dimension
Dimension
many:1 relationship
1-way filter direction

73. onprem.com
73

74. onprem.com
74
● General
○ Any information about the airline beyond the IATA 3-character code
○ Any information about airports (location, size, status, etc.)
○ Any information about individual airplanes beyond a six-digit Hex code identifying the transponder (which has data quality issues too)
○ "Clean" data -- We get negative velocities, negative altitudes, garbled callsigns, missing timestamps, every single field can have a
problem
● Points vs Legs/Flights
○ Any simple notion of how to group points into a "leg”
○ Any points over the open ocean (transponder network is land-based, not satellite)
■ This makes it harder to group points into a “leg” – we can’t assume that points for a single leg will be close time-wise to each
other
○ Points are much more reliable and consistent in some areas of the world than others. Continental US very good. Mexico and China,
for example, are very sparse.
○ Sometimes points over land in US are missing for certain areas too (military/FCC/other reasons? Area 51?)
● Legs/Flights
○ Any notion of how multiple legs for a multi-leg flight with a single callsign relate to each other
○ Any simple way to separate commercial, cargo, private, charter, and military flights from each other
○ Takeoff or landing airport for a given flight (even after we can group points together to a leg)
○ Any notion of a flight schedule for the future – having a list of legs/flights at particular times in the past is not simple to extrapolate into
the future