JVM startup issues, Platform Engineering, AI, GenAI, LLM, Watsonx

1. Emily Jiang, Java Champion, FBCS
IBM, STSM, Cloud Native Architect and Advocate
19th December 2023
Java Developers, What Lies
Ahead?

2. Recent Success
– Application Starup problems

3. JVM Spikes
3
0
50
100
150
200
250
300
350
400
0 30 60 90
CPU
utilization
(%)
Time (sec)
Daytrader7 CPU consumption
CPU spikes caused
by JIT compilation
0
100000
200000
300000
400000
500000
600000
0 30 60 90
Resident
set
size
(KB)
Time (sec)
Daytrader7 memory footprint
Footprint spikes caused
by JIT compilation
Main issues:
• Need to over-provision to avoid OOM
• Very hard to do – JVMs have a non-deterministic
behavior
Main issues:
• Slow start-up and ramp-up times
• CPU spikes can cause auto-scaler to incorrectly launch
additional instances

4. Solutions
– Get rid of JVM
– GraalVM
– Rework JVM using Checkpoint Install in Userspace (CRIU)
– Coordinated Restore at Checkpoint (CRaC)
– Liberty InstantOn

5. Liberty + Semeru InstantOn : fast startup using Linux CRIU
Characteristics
Semeru
InstantOn
Semeru
JVM
Graal
Native
Full Java support Yes Yes No
‘Instant on’ Yes No Yes
High throughput Yes Yes No
Low memory (under load)
Yes Yes Yes?
Dev-prod parity Yes Yes No
Dev Build
Prod
Prod
Prod
checkpoint
restore
restore
restore
Target Liberty application container deployments
Start application containers in milliseconds, ideal
for serverless
Leverages Linux CRIU to perform checkpoint /
restore
Make it really easy to consume for a user of
Liberty containers
Available
in Liberty
GA
containers
now!
5
– https://openliberty.io/

6. What bothering us?

7. 7
Stressful Developers
– You Build It, You Run It
– Dev and Ops together
– IDE
– Build (Jenkins, Containers)
– Deploy (CI/CD)
– Maintain (SBOM, Vulnerabilities)

8. Platform Engineering
– https://www.beinformed.com/blog/gartners-top-10-tech-trends-2024-embracing-platform-engineering/

9. Platform Engineering
Platform engineering is the discipline of designing and
building toolchains and workflows that enable self-service
capabilities for software engineering organizations in the
cloud-native era.
Platform engineers provide an integrated product most often
referred to as an “Internal Developer Platform” covering the
operational necessities of the entire lifecycle of an
application.
- platformengineering.org
9
– https://www.gartner.com/en/articles/what-is-platform-engineering

10. Internal Developer Portal
10
An Internal Developer Platform (IDP) is built by a platform
team to build golden paths and enable developer self-
service.
– https://internaldeveloperplatform.org/what-is-an-internal-developer-platform/

11. Improving Developer Experience
1.“Improve developer experience by building internal
developer platforms to reduce cognitive load, developer toil
and repetitive manual work.”
2.“Platforms don’t enforce a specific toolset or approach – it
is about making it easy for developers to build and deliver
software while not abstracting away useful and differentiated
capabilities of the underlying core services”
3.“Platform engineering teams treat platforms as a product
(used by developers) and design the platform to be
consumed in a self-service manner.”
Source: A Software Engineering Leader’s Guide to Improving Developer Experience by Manjunath Bhat, Research VP, Software Engineering
Practice at Gartner. ( Full report behind paywall)
11
https://internaldeveloperplatform.org/platform-tooling/

12. 12
What Else can Improve
Developer Experience?

13. 13
Remove the tedious tasks!

14. AI
Artificial intelligence (AI) is the simulation of human intelligence in
machines that are programmed to think and act like humans.
Learning, reasoning, problem-solving, perception, and
language comprehension are all examples of cognitive abilities.

15. AI History 1956 John McCarthy held a
workshop at Dartmouth on
“artificial intelligence”
1957-1974 AI flourished
2011 IBM Watson won the
game Jeopardy!
Apple released Siri, the first
popular virtual assistant.
2015 OpenAI founded
2020, OpenAI announced
GPT-3
2021, OpenAI introduced
DALL-E
1950 Alan Turing “Computing
Machinery and Intelligence”
1997
IBM Deep Blue beat the world
chess champion, Gary
Kasparov.
15
– https://www.youtube.com/watch?v=056v4OxKwlI

16. GenAI
Generative AI (GenAI) refers to deep-learning models that can
generate high-quality text, images, and other content based on
the data they were trained on.
– https://research.ibm.com/blog/what-is-generative-AI

17. Generative AI
Anything
that creates
new content
Large language model
Great
at text
Foundation
model
Unlabeled
data
Transformer
ChatGPT
inspired interest…
But there is a
bigger concept,
e.g. GPT
Which will
change business
Building blocks of generative AI
– LLM =Data + Architecture + Training
– Foundation Models
– BERT
– GPT
– Claude
– Cohere
– Stable Diffusion

18. Retrieval-Augmented
Generation
Q&A
Summarization
Summarize info – meeting
minutes, etc
Content Generation
Create email, marketing
materials, etc.
Named Entity
Recognition
Produce audit data
Insight Extraction
Medical diagnose, etc.
Classification
Sort customer complainants,
security vulnerability
classification, etc.
The most common
generative AI tasks
implemented today

19. Applications of Foundation Models
Foundation Model AI system Applications
LaMDA (Google) Bard (Google) AI chat
GPT-3.5 (OpenAI) ChatGPT (OpenAI) AI Chat
GPT-3 (OpenAI) DataCamp AI Assistant Code generation
Codex (OpenAI) GitHub copilot (Microsoft) Code generation
AudioLM (Google) MusicLM (Google) Create Music
BLOOM (Hugging Face) Use directly Mutiple NLP tasks. Trained in 46
languages and 13 programming
languages.
LLaMA (Meta) Use directly AI research
DALL-E 2 (OpenAI) Use directly Image creation
– https://www.datacamp.com/blog/what-are-foundation-models

20. Some GenAI tools
Chatbot
– Anthropic’s Claude 2
– Google’s Bard
– Meta AI’s Hugging Face Llama 2 Chat
– Microsoft’s Bing Chat
– OpenAI’s ChatGPT
AI code assistant
Github Copilot
Amazon CodeWisperer
Divi AI
Tabnine
Replit
Sourcegraphy Cody
20
– AI Code Assistants
– https://www.elegantthemes.com/blog/wordpress/best-ai-coding-assistant#4-tabnine
– https://www.youtube.com/watch?v=TXtnFw9eDmM

21. Tasks AI will do for us
• Generate code snippet
• Create tests
• Debugging
• Code review
• Refactoring
21

22. Issues related to AI
• License?
• Audit?
• Potentially generate bad code
• Security risk
• Lack of innovation
22

23. Bias
Believe that Generative
AI will propagate
established biases.
Source: IBM IBV “Generative AI: The state of the market”, June 2023
Agree Neutral Disagree
46%
Explainability
Believe decisions made by
Generative AI are not
sufficiently explainable.
Ethics
Concerned about the safety
and ethical aspects of
Generative AI.
Trust
Believe Generative AI
cannot be trusted.
46%
48% 42%
Business leaders face challenges in scaling AI across the enterprise with trust
80% of surveyed business leaders see at least one of these ethical issues as a major concern

24. Introducing…
watsonx

25. watsonx
Scale and
accelerate the
impact of AI with
trusted data.
A next generation enterprise
studio for AI builders to build,
train, validate, tune, and deploy
both traditional machine learning
and new generative AI
capabilities powered by
foundation models. It enables
you to build AI applications in a
fraction of the time with a
fraction of the data.
Fit-for-purpose data store, built on
an open lakehouse architecture,
supported by querying, governance
and open data formats to access
and share data.
End-to-end toolkit for AI
governance across the entire model
lifecycle to accelerate responsible,
transparent, and explainable AI
workflows
watsonx.ai
Build, train, validate, tune and
deploy AI models
watsonx.data
Scale AI workloads, for all
your data, anywhere
watsonx.governance
Accelerate responsible,
transparent and explainable AI
workflows
The platform
for AI and data

26. IBM foundation
models
All IBM AI models trained on
curated, enterprise-focused
data lake. IBM AI models
include:
• Slate (multilingual,
distilled, 153 million,
encoder-only); Fine
tuning required to
support extract and
classify language tasks
• Granite series models
(13b parameters,
.instruct and .chat,
decoder-only); Supports
all 5 NLP tasks
And more coming soon!
Open-source large
language models
5 open-source models are
sourced from Hugging Face
including:
• flan-ul2 (20b parameters,
encoder/decoder);
Supports all 5 NLP tasks
Third-party models added:
• StarCoder (15.5b
parameters, decoder-
only); CodeGen model
• Llama 2-chat (70b
parameters, decoder-
only); Supports all 5 NLP
tasks
watsonx.ai: Foundation Model Library
Explore the different foundation models offered in watsonx.ai to cover a range of enterprise use cases

27. Models available in watsonx.ai
granite.13b
13 billion params
decoder only
Generate
Extract
Summarize
Classify
Q&A
Class 3
8k
IBM Model
Instruct
Why Me:
Built on enterprise-
relevant datasets;
IP protections
flan-ul2-20b
20 billion params
encoder/decoder
Generate
Extract
Summarize
Classify
Q&A
Class 3
4k
Open Source
Instruct
Why Me:
Flexibility
gpt-neox-20b
20 billion params
decoder only
Generate
Q&A
Class 3
8k
Open Source
Why Me:
Special Characters
Context Length
mt0-xxl-13b
13 billion params
encoder/decoder
Generate
Extract
Summarize
Classify
Q&A
Class 2
4k
Open Source
Instruct
Why Me:
Multi-Lingual Model
100+ languages
flan-t5-xxl-11b
11 billion params
encoder/decoder
Generate
Summarize
Classify
Q&A
Class 2
4k
Open Source
Instruct
Why Me:
Medium Instruct
mpt-instruct2-7b
7 billion params
decoder only
Generate
Q&A
Class 1
2k
Open Source
Instruct
Why Me:
Small Instruct
llama2
70 billion params
decoder only
Generate
Extract
Summarize
Classify
Q&A
Class 3
4k
3rd Party
Instruct
Why Me:
Personality
starcoder
15.5 billion params
decoder only
CodeGen
Class 2
8k
3rd Party
Why Me:
Code
Note: Llama 2 and StarCoder have non-standard open-source terms with additional
Acceptable Use Policies

28. CLIENT BRIEFING
Discussion and custom demonstration
of IBM’s generative AI watsonx point-
of-view and capabilities. Understand
how watsonx.ai can be leveraged in
your AI strategy.
PILOT PROGRAM
watsonx.ai pilot developed with
IBM AI engineers. Prove watsonx.ai
value for the selected use case(s)
with a plan for adoption.
Three ways to get started with watsonx.ai today
IBM’s investment in partnering with you
FREE TRIAL
Experience watsonx.ai and test
out core capabilities yourself
with a free trial
Try our free trial 2-4 hours
Onsite or virtual
1-4 weeks
– https://www.ibm.com/watsonx

29. IBM’s AI is based on the
best open technologies available
IBM’s AI is transparent,
responsible, and governed
Open
Empowering
Trusted
IBM’s AI is for value creators,
not just users
Targeted IBM’s AI is designed for enterprise
and targeted at business domains
IBM Watsonx

30. Crucial skills for Java
Developers
• Focus on the architecture
• Innovation
• Serviceability
30

31. What jobs at risk because of AI
• Data Entry Clerk
• Telemarketer
• Factory Worker
• Cashier
• Driver
• Travel Agent
• Translator
• …
31
– https://unmudl.com/blog/careers-replaced-by-ai
– Developer not going to be replaced By AI

32. New Jobs Created by AI
• Prompt engineer?
• LLM Model Trainer?
• …
32

33. What should we do?
• Embrace AI
• Stay ahead of the new Tech
• Learn new skills
• Focus more on Architecture,
Innovation, etc.
33

34. 34
Thank You
Emily Jiang
IBM, Cloud Native Architect and Advocate
emijiang@uk.ibm.com
X/LinkedIn: @emilyfhjiang