1. The document discusses using machine learning to help optimize generic compilers by learning better heuristics and parameters from large amounts of data. 2. Early works tried feature engineering and traditional machine learning models with limited success due to the large design space. 3. DeepTune used an LSTM model with embedding layers to directly learn optimizations from program code and outperformed baselines on two hardware targets. 4. CompilerGym is proposed as a framework to implement and evaluate machine learning-based compiler optimizations as reinforcement learning agents interacting with a compiler environment.

1. CONFIDENTIAL
ML also helps Generic Compiler ?
Ryo Takahashi

2. 2
Domain Specific or Generic Compiler ?
Target of CompilerGym
Generic Compiler
• Optimize arbitrary programs
• Generate executable machine code
Domain Specific Compiler
• Pursue performance/productivity
of programs in a specific domain
stencil computation
(e.g. neural networks)
statical model (MCMC)

3. 3
How ML is involved in generic compilation ?
● Difficulty in compiler development
l Middle-end consists of hundreds of heuristics and
parameters invented by experts (e.g. JF Bastien-san)
l The current architecture is not scalable for development,
thus it tends to fall into sub-optimal performance (even LLVM !!)
Find better optimization heuristics and parameters by ML !!
clang
swiftc
rustc
C/C++/Objective-C
Swift
Rust
Front-end
LLVM IR
Optimization
(IR transformation)
x86-64
ARM
PowerPC
codegen
codegen
codegen
Back-end
Middle-end
e.g. loop unrolling, tiling,vectorization,
thread coarsening, device mapping, …

4. 4
Early works
● Since [Calder+ TOPLAS ’97], many academic researchers had tried ML
l Feature engineering for all of (Optimization Decisions) x (Targets) is infeasible
Step 1: Feature engineering
• case: unrolling factors
Still unacceptable engineering cost ...
Step 2: Predict the best decision
by traditional ML models
• e.g. decision tree

5. 5
DeepTune [Commins+ PaCT ’17 Best Paper!!]
● Use LSTM + Embedding layers for feature extraction
• Feed (almost) raw OpenCL code
• Handle the encoded sequence like NLP
• Make an optimzation decision with Affine
• Train modes for each target
Chris Cummns-san
(Facebook Researcher)

6. 6
Evaluation results
● Apply DeepTune to
device mapping decision
l CPU/GPU
● Outperformed the baselines
in the 2 platforms
l AMD/NVIDIA
Expert’s feture choice + decision tree
Static mapping

7. 7
After all, what is CompilerGym ?
● Background
l Compiler researchers see compiler optimizaiton tasks
as environements for RL [Leather+ FDL’20]
● CompilerGym
l Uses the OpenAI Gym to expose the “agent-environment loop”
l Support users to implement their own optimizers as RL agents
l Provides features extractors
and benchmark datasets
l Expose compiler APIs
state IR
action • IR transformation
• contex change
reward • speed up
• codesize reduction
DeepTune is
just this module

8. 8
Summary
● Finally, ML-based generic compiler optimization is
becoming less of a niche academic discipline !?
● However, full-scale implementation involving industry/OSS
has only just begun
● Let’s keep eye on activities of Chris-san and his ex-supervisor’s labo !!
(they have collaboration with ARM, Microsoft Research, and so on)