How WebAssembly can be used to optimize and accelerate Large Language Models Inference in the Cloud.

1. Samy Fodil
Founder & CEO of Taubyte
WebAssembly
is Key to Better
LLM
Performance

2. Samy Fodil
Founder & CEO of Taubyte

3. Taubyte
An Open Source Cloud
Platform on Autopilot, where
coding in local environment
equals scaling to global
production. 🚀
Local Coding
Global Production

4. Taubyte Locally

5. Taubyte In Prod

6. https://tau.how

7. LLMs
Generative AI

8. INPUT
Large Language Models
OUTPUT

9. Proprietary
Open Source
✅Train
✅Host
Large Language Models

10. LLM Training

11. LLM Inference
📦Huge PyTorch Images
🔗Complex Dependencies
🔒Lock GPU ressources
🗿Primitive Orchestration
Problems

12. 🪽Lightweight
🪣Sand-boxed
⚡Easy to orchestrate
LLM Inference
WASM to the rescue

13. 🪽 Lightweight
<2MB ~4GB

14. 🪽 Lightweight
⚡Fast to provision
🌱Cheap to cache
Closer to Data (Data Gravity) &
User (Edge Computing)

15. 🪣Sand-boxed
🛡️Secure
🕹️Interfaces
Which means No, Restricted, Virtual or
Mocked: Networking, Filesystem and more

16. LLM Inference
You can use transformers
directly from python. This
will result in PyTorch and
other dependencies.
📦Huge PyTorch Images
🔗Complex Dependencies
🔒Lock GPU ressources

17. LLM Inference
You can for example use onnx, llama.cpp, candle (which
wrapps llama.cpp) or TensorRT-LLM and have a lower foot
print. But it comes with a few challenges:
🔒Lock GPU ressources
🧠Way harder to implement

18. Scaling LLM Inference (lvl1)
🎹Orchestration
🌱Caching

19. Inference Engines for LLM
🤹Load balancing?
🚦Routing?

20. With WASM
If we made LLM and AI, in general, available through a set of
common host calls we can combine benfits of
Inference Engines
🎹Orchestration
🌱Caching
WebAssembly
🪽Lightweight
🪣Sand-boxed

21. 🤹Load balancing?
🚦Routing?

22. github.com/taubyte/tau
Will provide:
🤹Load-balancing
🚦Routing
🎹Orchestration
It’ll also provide abstractions so what’s built locally will work
in production with no changes.

23. APP HTTP Inference Engine
HTTP
HTTP
WASM
MODULE
HOST CALL
HOST CALL
HOST CALL
HTTP
HTTP
Inference
Host
Module
The idea
1️⃣
2️⃣

24. Implementation of 1️⃣
tau implement a protocols called `gateway` that will
determine what host will be best suited to serve the request
based on:
WebAssembly caching and dependency modules
availability (including host modules)
Host Module Resource availability
Host Resource availability
Other constraints defined by developer like data gravity

25. Node Running
Gateway Proto
Gateway Protocol
Serving Node
(Substrate)
Serving Node
(Substrate)
Serving Node
(Substrate)
Satellite
(i.e. LLM Inference)
Satellite
Satellite
MUXED
TUNNEL
MUXED TUNNEL
M
UXED
TUN
N
EL
ORBIT PROTO
ORBIT PROTO
ORBIT PROTO

26. Host Module Resource
Availability
This is still to be implemented feature that will ask Host
Module to provide basic metrics for the gateway:
Caching score. Example: is the particular model loaded.
Resources Availability Score. Example: Is there enough
GPU mem to spin-up the model
Queue Score. Example: If Host Module uses queues, how
filled is the queue.

27. github.com/taubyte/vm-orbit
Extending WebAssembly Runtime in a secure way.
orbit
external
process
PROXY WASM CALL
PROXY ACCESS TO MODULE MEMORY

28. Example

29. github.com/ollama-cloud
trunings ollama into a satellite

30. Compile llama.cpp
Build plugin

31. Install dreamland
Start local Cloud
Attach plug-in

32. Login to Local Cloud

33. Create a project

34. Create a function

35. Call Generate
Stream Tokens

36. No SDK!
Check my previous project
github.com/samyfodil/taubyte-llama-satellite
Which actually has a nice SDK!

37. Trigger the function

38. - Copy ollama plugin to /tb/plugins
- Add it to config
In production

39. Your Application is Live!

40. taubyte/dllama
Ready for Cloud with better backends
Always local friendly!

41. Thanks!