This paper proposes a serverless platform for building and operating edge AI applications. We analyze edge AI use cases to illustrate the challenges in building and operating AI applications in edge cloud scenarios. By elevating concepts from AI lifecycle management into the established serverless model, we enable easy development of edge AI workflow functions. We take a deviceless approach, i.e., we treat edge resources transparently as cluster resources, but give developers fine-grained control over scheduling constraints. Furthermore, we demonstrate the limitations of current serverless function schedulers, and present the current state of our prototype.

1. TU Wien, Vienna Austria
Distributed Systems Group
https://dsg.tuwien.ac.at
Thomas Rausch @thrauat
Waldemar Hummer
Vinod Muthusamy
Alexander Rashed
Schahram Dustdar
Towards a Serverless Platform for Edge AI
IBM Research AI
HotEdge’19, Renton, WA

2. 2
Drone
With Accelerator
Microsoft Build 2018 // Vision Keynote: https://www.youtube.com/watch?v=rd0Rd8w3FZ0

3. 3
Edge AI Accelerators
Google Edge TPU
NVIDIA Jetson
Intel
Neural Compute Stick
Baidu Kunlun
Microsoft
Project BrainWave
Huawei Atlas

4. 4
AI Operationalization
Hummer et al., ModelOps: Cloud-based Lifecycle Management for Reliable and Trusted AI. IC2E’19.
Process Train Validate Servee
Model
Runtime
Monitoring
Data
Perf.
Process Train Validate Serve
Object
Store
Compute
Cluster
Learning
Cluster
Read
Data
Train
Model
Write
Model
Data
Asset
Trained
Model
ModelOps Platform

5. 5
Serverless Model
{} Event (Request)
Trigger
Node
λ
λ
λ
λ
λ
λ
λ
λ
λ
Resource
λ
def handle(req):
s3 = boto3.client('s3')
with open(tmpfile, 'wb') as f:
s3.download_fileobj('bucket', req['obj'], f)
data = numpy.load(f)
m = train_model(data, req['train_params'])
s3.upload_fileobj(serialize(m), 'bucket', 'model'])
# ...
λλ
Function
Scheduler
Cloud Platform

6. 6
Deviceless Model
{} Event (Request)
Trigger
λλ
Function
Scheduler
??
Edge Cloud
Edge
Edge Cloud
Platform
λ
def handle(req):
s3 = boto3.client('s3')
with open(tmpfile, 'wb') as f:
s3.download_fileobj('bucket', req['obj'], f)
data = numpy.load(f)
m = train_model(data, req['train_params'])
s3.upload_fileobj(serialize(m), 'bucket', 'model'])
# ...

7. 7
 Data and Models as first-class citizens
 Model Selectors
 Policies
 Gates
AI Workflow
Programming Model
 Deviceless function scheduling
 Policy enactment
 Context awareness
 Data locality awareness
Execution Platform
A Serverless Platform for Edge AI
λ
λ

8. 8
@consumes.model(selector={
'type': 'image_classifier',
'data_tags': ['machine_x'],
'accuracy': '>=0.88'
})
def inference(model: Model, request):
data = request['input']
# data prep tasks
prediction = model.estimate(data)
@policy.deadline('2s')
@policy.fn(node = 'user_device',
capability = 'gpu')
@policy.data(network=['company_network'],
strict=True)
@consumes.data(
selector={'urn': 'mnist:data'},
holdout=0.2)
@produce.model(
type='classifier',
urn='mnist:model')
def train(data: Data, request) -> Model:
arr = data.to_ndarray()
return Model(train_model(arr))
@gate.bias(attribute = 'age',
predicate = '<0.8')
@gate.drift(metric = 'confidence',
predicate = '<0.2')
λ

9. 9
@consumes.model(selector={'urn': 'model:base'})
@consumes.data(batch = 100, selector=...)
@produces.model(type='regressor', urn='model:user:{usr}')
@policy.fn(node = 'local')
@policy.data(network = 'local', strict=True)
def refine(model: Model, data: Data):
ndarr = data.to_ndarray() # data artifact API
# transfer learning code
return refined_model
@consumes.model(selector={'urn': 'model:base'})
@consumes.data(batch = 100, selector=...)
@produces.model(type='regressor', urn='model:user:{usr}')
@policy.fn(node = 'local')
@policy.data(network = 'local', strict=True)
def refine(model: Model, data: Data):
ndarr = data.to_ndarray() # data artifact API
# transfer learning code
return refined_model
Network (edge, private)
node:{user}
container
Network (cloud)
f(x)
model u
data
data locality node
model b
λ
Function preprocessor
Scheduler

10. 10
Data Locality Tradeoffs
Cluster Middleware Cluster Middleware Cluster Middleware Cluster Middleware
h
Data
proximity
Container
Image
Deploy the container image to the edge?
OR
Send the data to the cloud?
Edge

11. 11
Skippy
 Built on and Kubernetes
 Kubernetes daemon to discover node capabilities
 Custom Python-based Kubernetes scheduler
● Adds inter-node proximity and data locality as constraints
● Non-monolithic architecture
 Coming to GitHub soon™
λ

12. 12
Preprocess Train Inferenceλ λλ
Scheduler + Simulator: https://git.dsg.tuwien.ac.at/serverless-edge-ai/sched-sim
λ

13. 13
Dipl.-Ing. (MSc), BSc
Thomas Rausch
Research Assistant
TU Wien
Institute of Information Systems Engineering
Argentinierstrasse 8-194-02, Vienna, Austria
T: +43 1 58801-184838
E: trausch@dsg.tuwien.ac.at
https://dsg.tuwien.ac.at/staff/trausch
@consumes.model(selector={'urn': 'model:base'})
@consumes.data(batch = 100, selector=...)
@produces.model(type='regressor', urn='model:user:{usr}')
@policy.fn(node = 'local')
@policy.data(network = 'local', strict=True)
def refine(model: Model, data: Data):
ndarr = data.to_ndarray() # data artifact API
# transfer learning code
return refined_model
Network (edge, private)
node:{user}
container
Network (cloud)
f(x)
model u
data
data locality node
model b
λ
Function preprocessor
Scheduler
{} Event (Request)
Trigger
λλ
Function
Scheduler
Edge Cloud
Edge
Cloud
Platform
λ
def handle(req):
s3 = boto3.client('s3')
with open(tmpfile, 'wb') as f:
s3.download_fileobj('bucket', req['obj'], f)
data = numpy.load(f)
m = train_model(data, req['train_params'])
s3.upload_fileobj(serialize(m), 'bucket', 'model'])
# ...
λ

14. 14
Discussion
●
Correct level of abstraction?
●
API/SDK features?
●
Validation criteria?
●
Deviceless model (does it work?)
●
Transparent data management
●
Scheduler architecture
●
Request routing architecture
●
Proximity and bandwidth monitoring
●
Learning optimal placements
●
Model too high-level for scheduler
●
“Bring-your-own-device” will fail
Feedbacki Controversial pointsii
Open issuesiii Failure risksiv