Spark Summit, profile picture
Uploaded bySpark Summit
PDF, PPTX6,583 views

Analyzing Log Data With Apache Spark

The document discusses analyzing log data using Apache Spark. It covers challenges with log data like schema mediation and feature engineering to transform log records into vectors. It also discusses visualizing the structured data using dimensionality reduction techniques like PCA and self-organizing maps to find outliers. The document provides examples of analyzing log data to identify the most frequent error levels and applications generating logs.

Embed presentation

Download as PDF, PPTX
1 / 112
2 / 112
3 / 112
4 / 112
5 / 112
6 / 112
7 / 112
8 / 112
9 / 112
10 / 112
11 / 112
12 / 112
13 / 112
14 / 112
15 / 112
16 / 112
17 / 112
18 / 112
19 / 112
20 / 112
21 / 112
22 / 112
23 / 112
24 / 112
25 / 112
26 / 112
27 / 112
28 / 112
29 / 112
30 / 112
31 / 112
32 / 112
33 / 112
34 / 112
35 / 112
36 / 112
37 / 112
38 / 112
39 / 112
40 / 112
41 / 112
42 / 112
43 / 112
44 / 112
45 / 112
46 / 112
47 / 112
48 / 112
49 / 112
50 / 112
51 / 112
52 / 112
53 / 112
54 / 112
55 / 112
56 / 112
57 / 112
58 / 112
59 / 112
60 / 112
61 / 112
62 / 112
63 / 112
64 / 112
65 / 112
66 / 112
67 / 112
68 / 112
69 / 112
70 / 112
71 / 112
72 / 112
73 / 112
74 / 112
75 / 112
76 / 112
77 / 112
78 / 112
79 / 112
80 / 112
81 / 112
82 / 112
83 / 112
84 / 112
85 / 112
86 / 112
87 / 112
88 / 112
89 / 112
90 / 112
91 / 112
92 / 112
93 / 112
94 / 112
95 / 112
96 / 112
97 / 112
98 / 112
99 / 112
100 / 112
101 / 112
102 / 112
103 / 112
104 / 112
105 / 112
106 / 112
107 / 112
108 / 112
109 / 112
110 / 112
111 / 112
112 / 112
Analyzing log data with Apache Spark William Benton Red Hat Emerging Technology
BACKGROUND
Challenges of log data
Challenges of log data SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
Challenges of log data 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
Challenges of log data postgres httpd syslog INFO INFO WARN CRIT DEBUG INFO GET GET GET POST WARN WARN INFO INFO INFO GET (404) INFO (ca. 2000)
Challenges of log data postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN (ca. 2016)
Challenges of log datapostgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN How many services are generating logs in your datacenter today?
DATA INGEST
Collecting log data collecting Ingesting live log data via rsyslog, logstash, fluentd normalizing Reconciling log record metadata across sources warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
Collecting log data warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
Collecting log data warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
Schema mediation
Schema mediation
Schema mediation
Schema mediation timestamp, level, host, IP addresses, message, &c. rsyslog-style metadata, like app name, facility, &c.
logs .select("level").distinct .map { case Row(s: String) => s } .collect Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … debug, notice, emerg, err, warning, crit, info, severe, alert
Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert
Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert This class must be declared outside the REPL!
FEATURE ENGINEERING
From log records to vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010
From log records to vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010 red pancakes orange waffles -> 00010000 -> 00101000
Similarity and distance
Similarity and distance
Similarity and distance (q - p) • (q - p)
Similarity and distance pi - qi i=1 n
Similarity and distance pi - qi i=1 n
Similarity and distance p • q p q
WARN INFO INFOINFO WARN DEBUGINFOINFOINFO WARN WARNINFO INFO INFO WAR INFO INFO Other interesting features host01 host02 host03
INFO INFOINFO DEBUGINFOINFO WARNNFO INFO INFO WARN INFO INFOINFO INFO INFO INFOINFOINFO INFOINFO INFO INFO INFO INFO WARN DEBUG Other interesting features host01 host02 host03
INFO INFOINFO INFO INFO INFOINFO INFO INFO INFO WARN DEBUG WARN INFO INFO EBUG WARN INFO INFO INFO INFO INFO INFO INFO INFO WARN WARN INFO WARN INFO Other interesting features host01 host02 host03
Other interesting features : Great food, great service, a must-visit! : Our whole table got gastroenteritis. : This place is so wonderful that it has ruined all other tacos for me and my family.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes. See https://links.freevariable.com/nlp-logs/ for more!
VISUALIZING STRUCTURE and FINDING OUTLIERS
Multidimensional data
Multidimensional data [4,7]
Multidimensional data [4,7]
Multidimensional data [4,7] [2,3,5]
Multidimensional data [4,7] [2,3,5]
Multidimensional data [4,7] [2,3,5] [7,1,6,5,12,
 8,9,2,2,4, 7,11,6,1,5]
Multidimensional data [4,7] [2,3,5] [7,1,6,5,12,
 8,9,2,2,4, 7,11,6,1,5]
A linear approach: PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
A linear approach: PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
Tree-based approaches
Tree-based approaches yes no yes no if orange if !orange if red if !red if !gray if !gray
Tree-based approaches yes no yes no if orange if !orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
Tree-based approaches yes no yes no if orange if !orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
Self-organizing maps
Self-organizing maps
Finding outliers with SOMs
Finding outliers with SOMs
Finding outliers with SOMs
Finding outliers with SOMs
Finding outliers with SOMs
Outliers in log data
Outliers in log data 0.95
Outliers in log data 0.95 0.97
Outliers in log data 0.95 0.97 0.92
Outliers in log data 0.95 0.97 0.92 0.37 An outlier is any record whose best match was at least 4σ below the mean. 0.94 0.89 0.91 0.93 0.96
Out of 310 million log records, we identified 0.0012% as outliers.
Thirty most extreme outliers 10 Can not communicate with power supply 2. 9 Power supply 2 failed. 8 Power supply redundancy is lost. 1 Drive A is removed. 1 Can not communicate with power supply 1. 1 Power supply 1 failed.
SOM TRAINING in SPARK
On-line SOM training
On-line SOM training
On-line SOM training
On-line SOM training
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt at each step, we update each unit by adding its value from the previous step…
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt to the example that we considered…
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt scaled by a learning factor and the distance from this unit to its best match
On-line SOM training
On-line SOM training sensitive to learning rate not parallel sensitive to example order
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods)
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) update the state of every cell in the neighborhood of the best matching unit, weighting by distance
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) keep track of the distance weights we’ve seen for a weighted average
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) since we can easily merge multiple states, we can train in parallel across many examples
Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
driver (using aggregate) workers
driver (using aggregate) workers
driver (using aggregate) workers
driver (using aggregate) workers What if you have a 3 mb model and 2,048 partitions?
driver (using treeAggregate) workers
driver (using treeAggregate) workers
driver (using treeAggregate) workers
driver (using treeAggregate) workers
driver (using treeAggregate) workers
SHARING MODELS BEYOND SPARK
Sharing models class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here }
Sharing models class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here } case class FrozenModel(entries: Array[Double], /* ... */ ) { }
Sharing models case class FrozenModel(entries: Array[Double], /* ... */ ) { } class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here def freeze: FrozenModel = // ... } object Model { def thaw(im: FrozenModel): Model = // ... }
Sharing models import org.json4s.jackson.Serialization import org.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) }
Sharing models import org.json4s.jackson.Serialization import org.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) } Also consider how you’ll share feature encoders and other parts of your learning pipeline!
PRACTICAL MATTERS
Spark and ElasticSearch Data locality is an issue and caching is even more important than when running from local storage. If your data are write-once, consider exporting ES indices to Parquet files and analyzing those instead.
Structured queries in Spark Always program defensively: mediate schemas, explicitly convert null values, etc. Use the Dataset API whenever possible to minimize boilerplate and benefit from query planning without (entirely) forsaking type safety.
Memory and partitioning Large JVM heaps can lead to appalling GC pauses and executor timeouts. Use multiple JVMs or off-heap storage (in Spark 2.0!) Tree aggregation can save you both memory and execution time by partially aggregating at worker nodes.
Interoperability Avoid brittle or language-specific model serializers when sharing models with non-Spark environments. JSON is imperfect but ubiquitous. However, json4s will serialize case classes for free! See also SPARK-13944, merged recently into 2.0.
Feature engineering Favor feature engineering effort over complex or novel learning algorithms. Prefer approaches that train interpretable models. Design your feature engineering pipeline so you can translate feature vectors back to factor values.
@willb • willb@redhat.com
 https://chapeau.freevariable.com THANKS!

More Related Content

PDF
set zeroth review blockchain electoral system.pdf
bySudamaSharma7
 
PDF
Apache Spark Overview part1 (20161107)
bySteve Min
 
PDF
OC Big Data Monthly Meetup #5 - Session 2 - Sumo Logic
byBig Data Joe™ Rossi
 
PPTX
Build a DataWarehouse for your logs with Python, AWS Athena and Glue
byMaxym Kharchenko
 
PDF
Polyglot Persistence in the Real World: Cassandra + S3 + MapReduce
bythumbtacktech
 
PDF
Tues 115pm cassandra + s3 + hadoop = quick auditing and analytics_yazovskiy
byAnton Yazovskiy
 
PDF
OSMC 2025: Easy logging refinement with FlowG by David Delassus.pdf
byNETWAYS
 
PDF
Scaling Security Threat Detection with Apache Spark and Databricks
byDatabricks
 
set zeroth review blockchain electoral system.pdf
bySudamaSharma7
 
Apache Spark Overview part1 (20161107)
bySteve Min
 
OC Big Data Monthly Meetup #5 - Session 2 - Sumo Logic
byBig Data Joe™ Rossi
 
Build a DataWarehouse for your logs with Python, AWS Athena and Glue
byMaxym Kharchenko
 
Polyglot Persistence in the Real World: Cassandra + S3 + MapReduce
bythumbtacktech
 
Tues 115pm cassandra + s3 + hadoop = quick auditing and analytics_yazovskiy
byAnton Yazovskiy
 
OSMC 2025: Easy logging refinement with FlowG by David Delassus.pdf
byNETWAYS
 
Scaling Security Threat Detection with Apache Spark and Databricks
byDatabricks
 

Similar to Analyzing Log Data With Apache Spark

PDF
Monitoring Error Logs at Databricks
byDatabricks
 
PPTX
Observability - the good, the bad, and the ugly
byAleksandr Tavgen
 
PDF
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
byEuangelos Linardos
 
PDF
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
byNETWAYS
 
PDF
Hadoop, hive和scribe在运维方面的应用
byxshadowxc
 
PDF
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
byVictoriaMetrics
 
PDF
Keeping the fun in functional w/ Apache Spark @ Scala Days NYC
byHolden Karau
 
PDF
Data Science
byAhmet Bulut
 
PDF
A Practical Guide to Anomaly Detection for DevOps
byBigPanda
 
PDF
More Than Just The Tip Of The Iceberg
byMichal Gancarski
 
PPTX
System Logs Anomaly Detection Using Deep Learning
bySrinivas Kanakala
 
PDF
Cassandra Talk: Austin JUG
byStu Hood
 
PDF
Fluentd meetup #3
byTreasure Data, Inc.
 
PPTX
Challenges of monitoring distributed systems
byNenad Bozic
 
PPTX
Analyzing-Logs-of-Distributed-System-Components-Using-AIML-Techniques.pptx
byRonitSharma65
 
PDF
Data Science Lab Meetup: Cassandra and Spark
byChristopher Batey
 
PDF
A fast introduction to PySpark with a quick look at Arrow based UDFs
byHolden Karau
 
PDF
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
PPTX
Observability – the good, the bad, and the ugly
byTimetrix
 
PDF
paper
byLukas Klingsbo
 
Monitoring Error Logs at Databricks
byDatabricks
 
Observability - the good, the bad, and the ugly
byAleksandr Tavgen
 
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
byEuangelos Linardos
 
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
byNETWAYS
 
Hadoop, hive和scribe在运维方面的应用
byxshadowxc
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
byVictoriaMetrics
 
Keeping the fun in functional w/ Apache Spark @ Scala Days NYC
byHolden Karau
 
Data Science
byAhmet Bulut
 
A Practical Guide to Anomaly Detection for DevOps
byBigPanda
 
More Than Just The Tip Of The Iceberg
byMichal Gancarski
 
System Logs Anomaly Detection Using Deep Learning
bySrinivas Kanakala
 
Cassandra Talk: Austin JUG
byStu Hood
 
Fluentd meetup #3
byTreasure Data, Inc.
 
Challenges of monitoring distributed systems
byNenad Bozic
 
Analyzing-Logs-of-Distributed-System-Components-Using-AIML-Techniques.pptx
byRonitSharma65
 
Data Science Lab Meetup: Cassandra and Spark
byChristopher Batey
 
A fast introduction to PySpark with a quick look at Arrow based UDFs
byHolden Karau
 
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
Observability – the good, the bad, and the ugly
byTimetrix
 
paper
byLukas Klingsbo
 

More from Spark Summit

PDF
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
PDF
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
bySpark Summit
 
PDF
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
bySpark Summit
 
PDF
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
bySpark Summit
 
PDF
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
bySpark Summit
 
PDF
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
bySpark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
PDF
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
bySpark Summit
 
PDF
Next CERN Accelerator Logging Service with Jakub Wozniak
bySpark Summit
 
PDF
Powering a Startup with Apache Spark with Kevin Kim
bySpark Summit
 
PDF
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
bySpark Summit
 
PDF
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
bySpark Summit
 
PDF
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
bySpark Summit
 
PDF
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
bySpark Summit
 
PDF
Goal Based Data Production with Sim Simeonov
bySpark Summit
 
PDF
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
bySpark Summit
 
PDF
Getting Ready to Use Redis with Apache Spark with Dvir Volk
bySpark Summit
 
PDF
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
bySpark Summit
 
PDF
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
bySpark Summit
 
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
bySpark Summit
 
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
bySpark Summit
 
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
bySpark Summit
 
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
bySpark Summit
 
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
bySpark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
bySpark Summit
 
Next CERN Accelerator Logging Service with Jakub Wozniak
bySpark Summit
 
Powering a Startup with Apache Spark with Kevin Kim
bySpark Summit
 
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
bySpark Summit
 
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
bySpark Summit
 
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
bySpark Summit
 
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
bySpark Summit
 
Goal Based Data Production with Sim Simeonov
bySpark Summit
 
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
bySpark Summit
 
Getting Ready to Use Redis with Apache Spark with Dvir Volk
bySpark Summit
 
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
bySpark Summit
 
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
bySpark Summit
 

Recently uploaded

PPTX
Update on TIMES Features Maintainer Report
byIEA-ETSAP
 
PDF
The Role of Data Science in Artificial Intelligence.pdf
byzameelmohd123pm
 
PDF
"AI Assistant Tools for Students: Boost Productivity with Claude, DeepSeek, A...
byCA Suvidha Chaplot
 
PDF
Solution Sheet / Objectives management (2026)
byOptymyze
 
PDF
Introduction to deep learning Important mmmmmmmm
byhossainmm2
 
PDF
Data Processing (2026)
byOptymyze
 
PPTX
07 ETSAP-TIMES_Technology-Database-Project-Proposal
byIEA-ETSAP
 
PDF
"Top Excel Tips and Tricks: Conditional Formatting, Sorting, and More - Learn...
byCA Suvidha Chaplot
 
PPTX
06 Proposal TIMES-TALKS A proposal for an IEA-ETSAP 12 part podcast series
byIEA-ETSAP
 
PDF
Modeling Styles of Vernacular Architecture using CIDOC CRM
bymagas4
 
PPTX
Improving tools for working with high resolution geographical and temporal da...
byIEA-ETSAP
 
PDF
Big Data Full Course Introduction to End.pdf
byJainnS
 
PPTX
TreatmentPlan of gingival diseases and options
byMohamadSaid29
 
PPTX
Business Ethics ppt (1).pptx for grade 12
bysuayanabegail
 
PDF
TOS in History.pdf ENGLISH macro.pdf mnbv
bykristinegorre7
 
PPTX
Recommended Practices for system studies with high shares of wind and solar
byIEA-ETSAP
 
PDF
UCL Research Group Invited Talk - QPP with Neural Models.pdf
bysuchanadatta3
 
PPTX
EMTECH-LESSON-3.pptxggggggggggggggggggggggg
byERICADEMESA
 
PDF
EXPLORING THE CARIBBEAN’S READINESS TO ADDRESS TECHNOLOGY-FACILITATED GENDER-...
byTaitu Heron
 
PDF
dell-ai-factory-with-nvidia-ebook_with.pdf
bytarimocarlos2014
 
Update on TIMES Features Maintainer Report
byIEA-ETSAP
 
The Role of Data Science in Artificial Intelligence.pdf
byzameelmohd123pm
 
"AI Assistant Tools for Students: Boost Productivity with Claude, DeepSeek, A...
byCA Suvidha Chaplot
 
Solution Sheet / Objectives management (2026)
byOptymyze
 
Introduction to deep learning Important mmmmmmmm
byhossainmm2
 
Data Processing (2026)
byOptymyze
 
07 ETSAP-TIMES_Technology-Database-Project-Proposal
byIEA-ETSAP
 
"Top Excel Tips and Tricks: Conditional Formatting, Sorting, and More - Learn...
byCA Suvidha Chaplot
 
06 Proposal TIMES-TALKS A proposal for an IEA-ETSAP 12 part podcast series
byIEA-ETSAP
 
Modeling Styles of Vernacular Architecture using CIDOC CRM
bymagas4
 
Improving tools for working with high resolution geographical and temporal da...
byIEA-ETSAP
 
Big Data Full Course Introduction to End.pdf
byJainnS
 
TreatmentPlan of gingival diseases and options
byMohamadSaid29
 
Business Ethics ppt (1).pptx for grade 12
bysuayanabegail
 
TOS in History.pdf ENGLISH macro.pdf mnbv
bykristinegorre7
 
Recommended Practices for system studies with high shares of wind and solar
byIEA-ETSAP
 
UCL Research Group Invited Talk - QPP with Neural Models.pdf
bysuchanadatta3
 
EMTECH-LESSON-3.pptxggggggggggggggggggggggg
byERICADEMESA
 
EXPLORING THE CARIBBEAN’S READINESS TO ADDRESS TECHNOLOGY-FACILITATED GENDER-...
byTaitu Heron
 
dell-ai-factory-with-nvidia-ebook_with.pdf
bytarimocarlos2014
 

Analyzing Log Data With Apache Spark

  • 1.
    Analyzing log datawith Apache Spark William Benton Red Hat Emerging Technology
  • 2.
  • 3.
  • 4.
    Challenges of logdata SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
  • 5.
    Challenges of logdata 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
  • 6.
    Challenges of logdata postgres httpd syslog INFO INFO WARN CRIT DEBUG INFO GET GET GET POST WARN WARN INFO INFO INFO GET (404) INFO (ca. 2000)
  • 7.
    Challenges of logdata postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN (ca. 2016)
  • 8.
    Challenges of logdatapostgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN How many services are generating logs in your datacenter today?
  • 9.
  • 10.
    Collecting log data collecting Ingestinglive log data via rsyslog, logstash, fluentd normalizing Reconciling log record metadata across sources warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
  • 11.
    Collecting log data warehousing Storingnormalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
  • 12.
    Collecting log data warehousing Storingnormalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
  • 13.
  • 14.
  • 15.
  • 16.
    Schema mediation timestamp, level,host, IP addresses, message, &c. rsyslog-style metadata, like app name, facility, &c.
  • 17.
    logs .select("level").distinct .map { caseRow(s: String) => s } .collect Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … debug, notice, emerg, err, warning, crit, info, severe, alert
  • 18.
    Exploring structured data logs .groupBy($"level",$"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert
  • 19.
    Exploring structured data logs .groupBy($"level",$"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert This class must be declared outside the REPL!
  • 20.
  • 21.
    From log recordsto vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010
  • 22.
    From log recordsto vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010 red pancakes orange waffles -> 00010000 -> 00101000
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
    WARN INFO INFOINFO WARNDEBUGINFOINFOINFO WARN WARNINFO INFO INFO WAR INFO INFO Other interesting features host01 host02 host03
  • 30.
    INFO INFOINFO DEBUGINFOINFO WARNNFO INFOINFO WARN INFO INFOINFO INFO INFO INFOINFOINFO INFOINFO INFO INFO INFO INFO WARN DEBUG Other interesting features host01 host02 host03
  • 31.
    INFO INFOINFO INFO INFO INFOINFO INFOINFO INFO WARN DEBUG WARN INFO INFO EBUG WARN INFO INFO INFO INFO INFO INFO INFO INFO WARN WARN INFO WARN INFO Other interesting features host01 host02 host03
  • 32.
    Other interesting features :Great food, great service, a must-visit! : Our whole table got gastroenteritis. : This place is so wonderful that it has ruined all other tacos for me and my family.
  • 33.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK.
  • 34.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers.
  • 35.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes.
  • 36.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes. See https://links.freevariable.com/nlp-logs/ for more!
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
    A linear approach:PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
  • 46.
    A linear approach:PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
  • 48.
  • 49.
    Tree-based approaches yes no yes no if orange if!orange if red if !red if !gray if !gray
  • 50.
    Tree-based approaches yes no yes no if orange if!orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
  • 51.
    Tree-based approaches yes no yes no if orange if!orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
    Outliers in logdata 0.95
  • 61.
    Outliers in logdata 0.95 0.97
  • 62.
    Outliers in logdata 0.95 0.97 0.92
  • 63.
    Outliers in logdata 0.95 0.97 0.92 0.37 An outlier is any record whose best match was at least 4σ below the mean. 0.94 0.89 0.91 0.93 0.96
  • 65.
    Out of 310million log records, we identified 0.0012% as outliers.
  • 68.
    Thirty most extremeoutliers 10 Can not communicate with power supply 2. 9 Power supply 2 failed. 8 Power supply redundancy is lost. 1 Drive A is removed. 1 Can not communicate with power supply 1. 1 Power supply 1 failed.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt
  • 75.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt at each step, we update each unit by adding its value from the previous step…
  • 76.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt to the example that we considered…
  • 77.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt scaled by a learning factor and the distance from this unit to its best match
  • 78.
  • 79.
    On-line SOM training sensitiveto learning rate not parallel sensitive to example order
  • 80.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods)
  • 81.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) update the state of every cell in the neighborhood of the best matching unit, weighting by distance
  • 82.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) keep track of the distance weights we’ve seen for a weighted average
  • 83.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) since we can easily merge multiple states, we can train in parallel across many examples
  • 84.
  • 85.
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.
  • 94.
    driver (using aggregate) workers Whatif you have a 3 mb model and 2,048 partitions?
  • 95.
  • 96.
  • 97.
  • 98.
  • 99.
  • 100.
  • 101.
    Sharing models class Model(privatevar entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here }
  • 102.
    Sharing models class Model(privatevar entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here } case class FrozenModel(entries: Array[Double], /* ... */ ) { }
  • 103.
    Sharing models case classFrozenModel(entries: Array[Double], /* ... */ ) { } class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here def freeze: FrozenModel = // ... } object Model { def thaw(im: FrozenModel): Model = // ... }
  • 104.
    Sharing models import org.json4s.jackson.Serialization importorg.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) }
  • 105.
    Sharing models import org.json4s.jackson.Serialization importorg.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) } Also consider how you’ll share feature encoders and other parts of your learning pipeline!
  • 106.
  • 107.
    Spark and ElasticSearch Datalocality is an issue and caching is even more important than when running from local storage. If your data are write-once, consider exporting ES indices to Parquet files and analyzing those instead.
  • 108.
    Structured queries inSpark Always program defensively: mediate schemas, explicitly convert null values, etc. Use the Dataset API whenever possible to minimize boilerplate and benefit from query planning without (entirely) forsaking type safety.
  • 109.
    Memory and partitioning LargeJVM heaps can lead to appalling GC pauses and executor timeouts. Use multiple JVMs or off-heap storage (in Spark 2.0!) Tree aggregation can save you both memory and execution time by partially aggregating at worker nodes.
  • 110.
    Interoperability Avoid brittle orlanguage-specific model serializers when sharing models with non-Spark environments. JSON is imperfect but ubiquitous. However, json4s will serialize case classes for free! See also SPARK-13944, merged recently into 2.0.
  • 111.
    Feature engineering Favor featureengineering effort over complex or novel learning algorithms. Prefer approaches that train interpretable models. Design your feature engineering pipeline so you can translate feature vectors back to factor values.
  • 112.