SlideShare a Scribd company logo
1 of 27
Download to read offline
JIT as a Service
Compiling for Runtimes in the Cloud
Mark Stoodley
Eclipse OMR and Eclipse OpenJ9 project lead
2
Important Disclaimers
§ THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
§ WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED.
§ ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
§ ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
§ IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
§ IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT
OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
§ NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
– CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS
OR THEIR SUPPLIERS AND/OR LICENSORS
Runtimes need to operate in small spaces
8 cores
8GB
1 core, 1GB
1 core, 1GB
1 core, 1GB
1 core, 1GB
1 core, 1GB
1 core, 1GB
1 core, 1GB
1 core, 1GB
1. Virtualization 2. Microservice Architecture
Monolithic
Application
High Input
Load
Service
A
Service
B
Service
C
Service
D
Service
E
Service
F
Possibly lower load connections
Many lighter-weight applications need to be allocated to many small VMs
3
Driving some interest in
Ahead of Time (AOT) compilers
even for ”dynamic” languages
4
5
AOT compiler JIT compiler
Runtime CPU cycles none considerable early on
Runtime memory none considerable early on
Ability to prove things excellent more limited
Performance ramp-up immediate takes time
Target environment have to choose CPU its running on
Profile based optimizations can use earlier run
data but awkward
collects and uses data
from current run
Speculative optimization can’t really afford yes, aggressively
AOT vs. JIT compilers for dynamic languages
AOT compilation has some advantages but
leaves significant performance on the table
by itself
6
What if…
• What if we could JIT compile out of process
• On another machine or even on a cluster of machines
• With independently deployed CPUs and memory
• Able to serve multiple applications and sharing code
• Optimizing with information collected from many runtime clients
7
Enter “JIT as a Service”
8
1. Move runtime costs to remote service
…can afford JIT even for very lightweight runtimes
2. Still running alongside applications
…so no loss in code performance
9
Yes! There will be latency!
But maybe not as bad as you think
10
The Basic JITaaS Architecture
• A client-server model
• Bidirectional communication
• Target method + metadata sent to
server
• Compiled code + metadata
returned to client
Client Server
Compilation begins
VM queries ×N
Compilation ends
Time
11
Two ongoing projects
1. Eclipse OMR
• Record JitBuilder calls on client, replay on server
2. Eclipse OpenJ9
• Compile Java methods from bytecodes
12
IBM ExtremeBlue student project over the summer :
1. Record JitBuilder API calls on the client side
2. Send that record to a JIT server
3. If no code generated already for the provided record:
4. Replay the recorded calls to JitBuilder implementation in the JIT server
5. Store generated code using the record as key
6. Send the code back to be installed in client’s code cache
7. Client can call the native code as a C function pointer
Created a client JIT for ultra simple (but Turing complete!) “BF” language
https://en.wikipedia.org/wiki/Brainf***
JitBuilder as a Service
Demo 1
JitBuilder as a Service
14
Eclipse OpenJ9x
Java JIT as a Service
15
8 Steps to a Remote Compiled Method
16
Client Server
Graphs & Numbers
• Throughput
• Memory Consumption
• Constrained environments
17
Throughput
18
0
2000
4000
6000
8000
10000
12000
14000
0 50 100 150 200 250 300
Pages/sec
Time (seconds)
AcmeAir Throughput
OpenJ9
JITaaS
Throughput Takeaways
• JIT as a service doesn’t have to sacrifice peak throughput
• Network latency (in aggregate) is tolerable
• Many parallel compilation requests overlap much of the cost
• Can cache many query results to avoid network round trips
19
Memory Consumption
20
0
50
100
150
200
250
300
350
0 50 100 150 200 250 300
Residentsetsize(MiB)
Time (seconds)
AcmeAir Footprint
OpenJ9
JITaaS Client
Memory Consumption Takeaways
• Dominant memory consumption is now application
• Size your containers to your apps, forget about the JIT
• Workload balancers will no longer “fooled” by spiky JIT activity
21
Demo 2
Java Jit as a Service
in a Constrained Environment
(256MB, ½ core)
22
Private
Future Work
• Close remaining performance gap
• Share compilations between clients
• Optimizations enabled through connection to multiple clients
• Build different kinds of remote JIT clients, not just for languages
• Merge hardened prototypes to OMR/OpenJ9 master branches
24
Shameless self plug: Whittier room at 3:30 today by Mark Stoodley
https://2018.splashcon.org/event/splash-2018-splash-i-oh-the-compilers-you-will-build-
All open and on the way to merged
• Eclipse OMR
• https://github.com/eclipse/omr/pull/3019 (JitBuilder record function)
• https://github.com/eclipse/omr/pull/3056 (JitBuilder replay function)
• https://eclipse-omr.slack.com
• Eclipse OpenJ9
• https://github.com/eclipse/openj9/tree/jitaas
• https://github.com/eclipse/openj9-omr/tree/jitaas
• https://openj9.slack.com
25
Questions?
26
Throughput in a Constrained Environment
27
0
200
400
600
800
1000
1200
1400
1600
0 100 200 300 400 500
Pages/sec
Time (seconds)
AcmeAir Throughput in a Micro-Container
(64 MiB, 0.5 CPU)
HotSpot
OpenJ9
JITaaS

More Related Content

Similar to Turbo2018 workshop JIT as a Service

Building Efficient Edge Nodes for Content Delivery Networks
Building Efficient Edge Nodes for Content Delivery NetworksBuilding Efficient Edge Nodes for Content Delivery Networks
Building Efficient Edge Nodes for Content Delivery NetworksRebekah Rodriguez
 
Bluemix Local – Relay Options and Challenges
Bluemix Local – Relay Options and Challenges Bluemix Local – Relay Options and Challenges
Bluemix Local – Relay Options and Challenges Eduardo Patrocinio
 
Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...
Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...
Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...Teodoro Cipresso
 
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...0xdaryl
 
Ingesting Data at Blazing Speed Using Apache Orc
Ingesting Data at Blazing Speed Using Apache OrcIngesting Data at Blazing Speed Using Apache Orc
Ingesting Data at Blazing Speed Using Apache OrcDataWorks Summit
 
2016-Automation-Summit_PA_SIMIT.pdf
2016-Automation-Summit_PA_SIMIT.pdf2016-Automation-Summit_PA_SIMIT.pdf
2016-Automation-Summit_PA_SIMIT.pdfLuisJonathanBahamaca
 
A Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and BeyondA Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and BeyondLaura Hood
 
Key Note Session IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...
Key Note Session  IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...Key Note Session  IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...
Key Note Session IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...Surekha Parekh
 
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...Indrajit Poddar
 
Servidor IBM zEnterprise BC12
Servidor IBM zEnterprise BC12Servidor IBM zEnterprise BC12
Servidor IBM zEnterprise BC12Anderson Bassani
 
Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...
Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...
Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...mfrancis
 
Dynamic and Scalable Systems Using WinCC OA
Dynamic and Scalable Systems Using WinCC OADynamic and Scalable Systems Using WinCC OA
Dynamic and Scalable Systems Using WinCC OADMC, Inc.
 
“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...
“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...
“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...Edge AI and Vision Alliance
 
FOSDEM2016 - Ruby and OMR
FOSDEM2016 - Ruby and OMRFOSDEM2016 - Ruby and OMR
FOSDEM2016 - Ruby and OMRCharlie Gracie
 
Developing Distributed Internet of Things Applications Made Easy with Concier...
Developing Distributed Internet of Things Applications Made Easy with Concier...Developing Distributed Internet of Things Applications Made Easy with Concier...
Developing Distributed Internet of Things Applications Made Easy with Concier...mfrancis
 
Enterprise Cloud with IBM & Chef (ChefConf 2013)
Enterprise Cloud with IBM & Chef (ChefConf 2013)Enterprise Cloud with IBM & Chef (ChefConf 2013)
Enterprise Cloud with IBM & Chef (ChefConf 2013)Michael Elder
 
Java Development on Bluemix
Java Development on BluemixJava Development on Bluemix
Java Development on BluemixRam Vennam
 
Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...
Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...
Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...Joao Galdino Mello de Souza
 
The Real World - Plugging the Enterprise Into It (nodejs)
The Real World - Plugging  the Enterprise Into It (nodejs)The Real World - Plugging  the Enterprise Into It (nodejs)
The Real World - Plugging the Enterprise Into It (nodejs)Aman Kohli
 

Similar to Turbo2018 workshop JIT as a Service (20)

Building Efficient Edge Nodes for Content Delivery Networks
Building Efficient Edge Nodes for Content Delivery NetworksBuilding Efficient Edge Nodes for Content Delivery Networks
Building Efficient Edge Nodes for Content Delivery Networks
 
Bluemix Local – Relay Options and Challenges
Bluemix Local – Relay Options and Challenges Bluemix Local – Relay Options and Challenges
Bluemix Local – Relay Options and Challenges
 
Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...
Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...
Innovate 2014: Get an A+ on Testing Your Enterprise Applications with Rationa...
 
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
JavaOne 2015 CON7547 "Beyond the Coffee Cup: Leveraging Java Runtime Technolo...
 
OpenDataPlane Project
OpenDataPlane ProjectOpenDataPlane Project
OpenDataPlane Project
 
Ingesting Data at Blazing Speed Using Apache Orc
Ingesting Data at Blazing Speed Using Apache OrcIngesting Data at Blazing Speed Using Apache Orc
Ingesting Data at Blazing Speed Using Apache Orc
 
2016-Automation-Summit_PA_SIMIT.pdf
2016-Automation-Summit_PA_SIMIT.pdf2016-Automation-Summit_PA_SIMIT.pdf
2016-Automation-Summit_PA_SIMIT.pdf
 
A Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and BeyondA Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
 
Key Note Session IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...
Key Note Session  IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...Key Note Session  IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...
Key Note Session IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...
 
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
 
Servidor IBM zEnterprise BC12
Servidor IBM zEnterprise BC12Servidor IBM zEnterprise BC12
Servidor IBM zEnterprise BC12
 
Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...
Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...
Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verb...
 
Dynamic and Scalable Systems Using WinCC OA
Dynamic and Scalable Systems Using WinCC OADynamic and Scalable Systems Using WinCC OA
Dynamic and Scalable Systems Using WinCC OA
 
“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...
“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...
“Smarter Manufacturing with Intel’s Deep Learning-Based Machine Vision,” a Pr...
 
FOSDEM2016 - Ruby and OMR
FOSDEM2016 - Ruby and OMRFOSDEM2016 - Ruby and OMR
FOSDEM2016 - Ruby and OMR
 
Developing Distributed Internet of Things Applications Made Easy with Concier...
Developing Distributed Internet of Things Applications Made Easy with Concier...Developing Distributed Internet of Things Applications Made Easy with Concier...
Developing Distributed Internet of Things Applications Made Easy with Concier...
 
Enterprise Cloud with IBM & Chef (ChefConf 2013)
Enterprise Cloud with IBM & Chef (ChefConf 2013)Enterprise Cloud with IBM & Chef (ChefConf 2013)
Enterprise Cloud with IBM & Chef (ChefConf 2013)
 
Java Development on Bluemix
Java Development on BluemixJava Development on Bluemix
Java Development on Bluemix
 
Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...
Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...
Modernização do Gerenciamento, Monitoramento e Provisionamento em Mainframes ...
 
The Real World - Plugging the Enterprise Into It (nodejs)
The Real World - Plugging  the Enterprise Into It (nodejs)The Real World - Plugging  the Enterprise Into It (nodejs)
The Real World - Plugging the Enterprise Into It (nodejs)
 

More from Mark Stoodley

Eliminating the Pauses in your Java Application
Eliminating the Pauses in your Java ApplicationEliminating the Pauses in your Java Application
Eliminating the Pauses in your Java ApplicationMark Stoodley
 
Oh the compilers you'll build
Oh the compilers you'll buildOh the compilers you'll build
Oh the compilers you'll buildMark Stoodley
 
Jit builder status and directions 2018 03-28
Jit builder status and directions 2018 03-28Jit builder status and directions 2018 03-28
Jit builder status and directions 2018 03-28Mark Stoodley
 
JavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVM
JavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVMJavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVM
JavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVMMark Stoodley
 
VMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developerVMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developerMark Stoodley
 
Under the Hood of the Testarossa JIT Compiler
Under the Hood of the Testarossa JIT CompilerUnder the Hood of the Testarossa JIT Compiler
Under the Hood of the Testarossa JIT CompilerMark Stoodley
 
Eclipse OMR: a modern toolkit for building language runtimes
Eclipse OMR: a modern toolkit for building language runtimesEclipse OMR: a modern toolkit for building language runtimes
Eclipse OMR: a modern toolkit for building language runtimesMark Stoodley
 

More from Mark Stoodley (7)

Eliminating the Pauses in your Java Application
Eliminating the Pauses in your Java ApplicationEliminating the Pauses in your Java Application
Eliminating the Pauses in your Java Application
 
Oh the compilers you'll build
Oh the compilers you'll buildOh the compilers you'll build
Oh the compilers you'll build
 
Jit builder status and directions 2018 03-28
Jit builder status and directions 2018 03-28Jit builder status and directions 2018 03-28
Jit builder status and directions 2018 03-28
 
JavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVM
JavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVMJavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVM
JavaOne 2017 - Mark Stoodley - Open Sourcing IBM J9 JVM
 
VMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developerVMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developer
 
Under the Hood of the Testarossa JIT Compiler
Under the Hood of the Testarossa JIT CompilerUnder the Hood of the Testarossa JIT Compiler
Under the Hood of the Testarossa JIT Compiler
 
Eclipse OMR: a modern toolkit for building language runtimes
Eclipse OMR: a modern toolkit for building language runtimesEclipse OMR: a modern toolkit for building language runtimes
Eclipse OMR: a modern toolkit for building language runtimes
 

Recently uploaded

How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 

Recently uploaded (20)

How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 

Turbo2018 workshop JIT as a Service

  • 1. JIT as a Service Compiling for Runtimes in the Cloud Mark Stoodley Eclipse OMR and Eclipse OpenJ9 project lead
  • 2. 2 Important Disclaimers § THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. § WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. § ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. § ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. § IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. § IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. § NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: – CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS
  • 3. Runtimes need to operate in small spaces 8 cores 8GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1 core, 1GB 1. Virtualization 2. Microservice Architecture Monolithic Application High Input Load Service A Service B Service C Service D Service E Service F Possibly lower load connections Many lighter-weight applications need to be allocated to many small VMs 3
  • 4. Driving some interest in Ahead of Time (AOT) compilers even for ”dynamic” languages 4
  • 5. 5 AOT compiler JIT compiler Runtime CPU cycles none considerable early on Runtime memory none considerable early on Ability to prove things excellent more limited Performance ramp-up immediate takes time Target environment have to choose CPU its running on Profile based optimizations can use earlier run data but awkward collects and uses data from current run Speculative optimization can’t really afford yes, aggressively AOT vs. JIT compilers for dynamic languages
  • 6. AOT compilation has some advantages but leaves significant performance on the table by itself 6
  • 7. What if… • What if we could JIT compile out of process • On another machine or even on a cluster of machines • With independently deployed CPUs and memory • Able to serve multiple applications and sharing code • Optimizing with information collected from many runtime clients 7
  • 8. Enter “JIT as a Service” 8
  • 9. 1. Move runtime costs to remote service …can afford JIT even for very lightweight runtimes 2. Still running alongside applications …so no loss in code performance 9
  • 10. Yes! There will be latency! But maybe not as bad as you think 10
  • 11. The Basic JITaaS Architecture • A client-server model • Bidirectional communication • Target method + metadata sent to server • Compiled code + metadata returned to client Client Server Compilation begins VM queries ×N Compilation ends Time 11
  • 12. Two ongoing projects 1. Eclipse OMR • Record JitBuilder calls on client, replay on server 2. Eclipse OpenJ9 • Compile Java methods from bytecodes 12
  • 13. IBM ExtremeBlue student project over the summer : 1. Record JitBuilder API calls on the client side 2. Send that record to a JIT server 3. If no code generated already for the provided record: 4. Replay the recorded calls to JitBuilder implementation in the JIT server 5. Store generated code using the record as key 6. Send the code back to be installed in client’s code cache 7. Client can call the native code as a C function pointer Created a client JIT for ultra simple (but Turing complete!) “BF” language https://en.wikipedia.org/wiki/Brainf*** JitBuilder as a Service
  • 14. Demo 1 JitBuilder as a Service 14
  • 15. Eclipse OpenJ9x Java JIT as a Service 15
  • 16. 8 Steps to a Remote Compiled Method 16 Client Server
  • 17. Graphs & Numbers • Throughput • Memory Consumption • Constrained environments 17
  • 18. Throughput 18 0 2000 4000 6000 8000 10000 12000 14000 0 50 100 150 200 250 300 Pages/sec Time (seconds) AcmeAir Throughput OpenJ9 JITaaS
  • 19. Throughput Takeaways • JIT as a service doesn’t have to sacrifice peak throughput • Network latency (in aggregate) is tolerable • Many parallel compilation requests overlap much of the cost • Can cache many query results to avoid network round trips 19
  • 20. Memory Consumption 20 0 50 100 150 200 250 300 350 0 50 100 150 200 250 300 Residentsetsize(MiB) Time (seconds) AcmeAir Footprint OpenJ9 JITaaS Client
  • 21. Memory Consumption Takeaways • Dominant memory consumption is now application • Size your containers to your apps, forget about the JIT • Workload balancers will no longer “fooled” by spiky JIT activity 21
  • 22. Demo 2 Java Jit as a Service in a Constrained Environment (256MB, ½ core) 22
  • 24. Future Work • Close remaining performance gap • Share compilations between clients • Optimizations enabled through connection to multiple clients • Build different kinds of remote JIT clients, not just for languages • Merge hardened prototypes to OMR/OpenJ9 master branches 24 Shameless self plug: Whittier room at 3:30 today by Mark Stoodley https://2018.splashcon.org/event/splash-2018-splash-i-oh-the-compilers-you-will-build-
  • 25. All open and on the way to merged • Eclipse OMR • https://github.com/eclipse/omr/pull/3019 (JitBuilder record function) • https://github.com/eclipse/omr/pull/3056 (JitBuilder replay function) • https://eclipse-omr.slack.com • Eclipse OpenJ9 • https://github.com/eclipse/openj9/tree/jitaas • https://github.com/eclipse/openj9-omr/tree/jitaas • https://openj9.slack.com 25
  • 27. Throughput in a Constrained Environment 27 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 Pages/sec Time (seconds) AcmeAir Throughput in a Micro-Container (64 MiB, 0.5 CPU) HotSpot OpenJ9 JITaaS