This document discusses using semantic graphs to optimize feature engineering for streaming timeseries data in industrial automation. It outlines challenges like high data volume, latency, and interdependence of streaming data. Semantic graphs can represent feature engineering computations as a directed graph, making relationships between concepts explicit. This allows for easier tracing of variables, code refactoring, and introducing parallelization by computing unrelated features simultaneously in separate threads. The document provides a simplified example of representing feature engineering equations as a semantic graph and how parallelization could be applied.

1. MACHINE LEARNING ON THE
RUN: OPTIMIZED FEATURE
ENGINEERING FOR STREAMING
TIMESERIES DATA IN
INDUSTRIAL AUTOMATION
- Mayank Prasoon

2. AGENDA
• Challenges with processing high volume streaming data for industrial
automation
• Using semantic graphs in feature engineering of streaming data
• Using graphs for model creation and processing
• Parallelization in computing semantic graphs
• Conclusions

3. Challenges in industry 4.0

4. About Quartic.ai and Myself
• Quartic.ai
• Industrial IoT Platform for Manufacturing Industries
• Used by Fortune 100 companies in Pharma, Mining, Power and F&B
• Data Processing:
• 10B + daily data points in the cloud
• 100B + daily data points on the edge
• Mayank Prasoon
• Software Developer and product enthusiast
• Quartic.ai
• https://www.linkedin.com/in/prasoonmayank/
• Ukulele enthusiast

5. LATENCY
• A typical manufacturing pipeline has 10s to 100s of different
machines with each having 100s of sensors to measure multiple
factors like pressure, temperature, pH levels and so on, with each
giving data at interval of a few milli seconds to seconds.
• Further the raw data has to be processed in near real time, to ensure
that the time lag between the raw data intake and the data
processing is minimal.
• The data processing involves multiple combination of logical,
arithmetic and binary operations on the incoming data stream.

6. INTERDEPENDENCE
• The processed data often has less number of dimensions compared
to the raw data. This is done by dimensionality reduction by
combining, removing or minimizing the input raw data. This makes
the feature engineering process dependent on the input tags.
• The sensors don’t give data at the same sampling rates. Also, the
sampling rates may vary at different times due to drift and noise.
• All these factors have to be taken into consideration when creating
the feature engineering process.

7. GRANULARITY
• The incoming data can be very granular which means the data
processing should be happening at similar time intervals.

8. DATA VOLUME
• The volume of data can easily be in billions data points, each day.
• Quartic’s ingestion and processing is ~10B data points per day

9. Using semantic graphs in feature engineering
of streaming data

10. What are semantic graphs?
• According to Wikipedia, a semantic graph is "a network that
represents semantic relationships between concepts...It is a directed
or undirected graph consisting of vertices, which represent concepts,
and edges, which represent semantic relations between concepts."

11. SEMANTIC GRAPHS IN THE CASE OF
INDUSTRIAL AUTOMATION

12. Scaled down problem statement:
• We get input from 5 different sensors(features here), which have to
be reduced to a single variable(Data here)
• Lets denote the variables as feature inputs as F1, F2, F3, F4, F5
• These are used to compute CF1, CF2, CF3 and CF4, which are further
used to compute CF`1, CF`2 and CF`3.
• We eventually calculate Data(T)
• The equations considered while calculating Data are shown in the
next slide

13. EQUATIONS CONSIDERED:
• CF1 = (F1 + F2 + F3) * Data(T-1)
• CF2 = (F2-F3)
• CF3 = F3 + CF2 + CF4
• CF4 = F4 – F5
• CF`1 = F2 + Data(T-1)
• CF`2 = if(CF`3) CF`2 else 0
• CF`3 = F4 + CF3
• Data(T) = f(CF`1, CF`2, CF`3, Data(T-1))

14. The above equations shown as a part of
semantic graph

15. SCALED DOWN VERSION FOR
APPLICATION IN INDUSTRIAL
AUTOMATION
F1
F2
F3
F4
F5
DATA(T-1)
CF1
CF2
CF3
CF4
CF`1
CF`2
CF`3
DATA(T)
F – Feature
CF – Computed Feature
CF` - Computed Feature 2
Data – Result calculated at t

16. What we noticed above?
• Each of the input variables are stored in nodes in the graph, while the
computations are stored as edges of the graph.
• It is relatively simpler to trace a particular computer variable back to
its origins.
• Advantages in code refactoring due to the semantic graph nature of
feature engineering.
• There is no recalculation of variables required.

17. INTRODUCING PARALLALIZATION

18. WHY?
• As can be seen in the above graph, some features don’t require the
calculation of other features. Also, in the practical case, the delay in
the streaming data for any feature should not affect the feature
calculation.

19. F1
F2
F3
F4
F5
DATA(T-1)
CF1
CF2
CF3
CF4
CF`1
CF`2
CF`3
DATA(T)
P1
P2

20. • In the above case, calculation of CF4 is completely independent of
calculation of CF1, CF2 and CF3. Thus, once, the calculation of CF1
starts, a new thread is spawned that checks for unrelated
computations as in CF4 and calculates them.
• Thus, as one moves further along the graph, one can always calculate
the unrelated variables by keeping a dictionary of all the calculated
and raw features, which is what we do in our application.

21. KEY TAKEAWAYS
• Difficulties in processing streaming data due to high granularity,
latency, interdependence and data volume
• Semantic graphs for feature engineering of streaming datasets
• Introducing parallelization in semantic graph

22. REFERENCES
• https://www.semanticscholar.org/paper/Mixed-Initiative-Feature-
Engineering-Using-Graphs-Atzm%C3%BCller-
Sternberg/31d92d052cedcef7ad6f3a04860c662a264c7e66
• https://www.sciencedirect.com/science/article/pii/01663615899011
15
• https://tdwi.org/articles/2018/11/09/adv-all-4-reasons-use-graphs-
optimize-machine-learning-data-engineering.aspx
• https://github.com/yahoo/graphkit

23. THANK YOU