This document discusses using Apache Spark to analyze web log data. Spark is well-suited for this task due to its performance on batch sizes smaller than total RAM and its high-level API. The document outlines parsing log lines, implementing a lambda architecture with Spark, configuring a Spark cluster with Linux containers, and techniques for managing Spark's memory usage such as caching frequently reused RDDs. Aggregation examples using groupBy and reduceByKey are also provided.

1. APACHE SPARK REX

2. ABOUT ME
Didier Marin
PhD in Computer Science (UPMC)
Machine Learning, Reinforcement Learning & Robotics
Co-founder of Heuritech
Likes functional programming and distributed computing

3. We develop tools to make sense from raw text data
Customer insight using the text of visited web pages

4. Data Analytics Platform
Qualify users using their web logs
50M lines/day
Match CRM and web data

6. WHY SPARK ?
Performance, in particular when
batch size < total RAM in cluster
More general than MR, high-level API
Extensions (ML, streaming) and
connectors (Cassandra)
Growing community

7. PARSING LOGS
defparseLine(line:String):
Either[ParsingError,LogData]=???
vallogs=sc.textFile("logfile").map(parseLine(_))
valvalidLogs=logs.flatMap(_.right.toOption)

8. LAMBDA ARCHITECTURE

9. IMPLEMENTATION

10. CLUSTER CONFIGURATION
LXC + salt
N containers : 1 master/executor + (N-1) executors
Cassandra node for each Spark executor
Using an "uber"-JAR to submit jobs
Sharing data through NFS

12. MANAGING SPARK'S MEMORY
Default: 40 % working memory, 60 % cache
20 % of cache used to unroll blocks
Explicit caching for huge RDDs we reuse:
validLogs.persist(StorageLevel.MEMORY_AND_DISK)
Partition tuning may be necessary (spark.default.parallelism)

13. AGGREGATION
valwords=sc.parallelize(List("a","b","a","c"))
words.groupBy(x=>x).mapValues(_.size).collect
//Array((a,2),(b,1),(c,1))
words.map(x=>(x,1)).reduceByKey(_+_).collect
//Array((a,2),(b,1),(c,1))

14. AGGREGATION
groupBy

15. see also &
AGGREGATION
reduceByKey
combineByKey foldByKey

16. Databricks knowledge base
Spark users mailing list
Parsing Apache logs with Spark (Scala)
USEFUL LINKS
github.com/databricks/spark-knowledgebase
apache-spark-user-list.1001560.n3.nabble.com
alvinalexander.com/scala/analyzing-apache-access-logs-files-
spark-scala

17. THANK YOU !
contact@heuritech.com