SlideShare a Scribd company logo
1 of 2
Download to read offline
A Co-Scheduling Framework for DNN
Models on Mobile and Edge Devices
With Heterogeneous Hardware
Abstract
With the emergence of more and more powerful chipsets and hardware and
the rise of Artificial Intelligence of Things (AIoT
bringing Deep Neural Network (DNN) models to empower mobile and edge
devices with intelligence such that they can support attractive AI applications
on the edge in a real
heterogeneous computational resources (such as CPU, GPU, DSP, etc) to
effectively and efficiently support concurrent inference of multiple DNN models
on a mobile or edge device, we propose a novel online Co
framework based on deep REinforcement Learning (
COSREL. COSREL has the following desirable features: 1) it achieves
significant speedup over commonly
computational resources on heterogeneous hardware; 2) it leverages
emerging Deep Reinforcement Learning (DRL) to make dynamic and wise
Scheduling Framework for DNN
Models on Mobile and Edge Devices
With Heterogeneous Hardware
With the emergence of more and more powerful chipsets and hardware and
the rise of Artificial Intelligence of Things (AIoT), there is a growing trend for
bringing Deep Neural Network (DNN) models to empower mobile and edge
devices with intelligence such that they can support attractive AI applications
on the edge in a real-time or near real-time manner. To leverage
ous computational resources (such as CPU, GPU, DSP, etc) to
effectively and efficiently support concurrent inference of multiple DNN models
on a mobile or edge device, we propose a novel online Co
framework based on deep REinforcement Learning (DRL), which we call
COSREL. COSREL has the following desirable features: 1) it achieves
significant speedup over commonly-used methods by efficiently utilizing all the
computational resources on heterogeneous hardware; 2) it leverages
cement Learning (DRL) to make dynamic and wise
Scheduling Framework for DNN
Models on Mobile and Edge Devices
With the emergence of more and more powerful chipsets and hardware and
), there is a growing trend for
bringing Deep Neural Network (DNN) models to empower mobile and edge
devices with intelligence such that they can support attractive AI applications
time manner. To leverage
ous computational resources (such as CPU, GPU, DSP, etc) to
effectively and efficiently support concurrent inference of multiple DNN models
on a mobile or edge device, we propose a novel online Co-Scheduling
DRL), which we call
COSREL. COSREL has the following desirable features: 1) it achieves
used methods by efficiently utilizing all the
computational resources on heterogeneous hardware; 2) it leverages
cement Learning (DRL) to make dynamic and wise
online scheduling decisions based on system runtime state; 3) it is capable of
making a good tradeoff among inference latency, throughput and energy
efficiency; and 4) it makes no changes to given DNN models, thus preserves
their accuracies. To validate and evaluate COSREL, we conduct extensive
experiments on an off-the-shelf Android smartphone with widely-used DNN
models to compare it with three commonly-used baselines. Our experimental
results show that 1) COSREL consistently and significantly outperforms all the
baselines in terms of both throughput and latency; and 2) COSREL is
generally superior to all the baselines in terms of energy efficiency.

More Related Content

Similar to A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices With Heterogeneous Hardware.pdf

Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
AIRCC Publishing Corporation
 
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...
ijcsit
 
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
AIRCC Publishing Corporation
 
A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING
A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING
A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING
mlaij
 
Grid computing [2005]
Grid computing [2005]Grid computing [2005]
Grid computing [2005]
Raul Soto
 
Network Centric to Data Centric Networks (Data Distribution Service)
Network Centric to Data Centric Networks (Data Distribution Service)Network Centric to Data Centric Networks (Data Distribution Service)
Network Centric to Data Centric Networks (Data Distribution Service)
Adil Khan
 

Similar to A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices With Heterogeneous Hardware.pdf (20)

Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
 
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...
 
3 rd International Conference on Signal Processing, VLSI Design & Communicati...
3 rd International Conference on Signal Processing, VLSI Design & Communicati...3 rd International Conference on Signal Processing, VLSI Design & Communicati...
3 rd International Conference on Signal Processing, VLSI Design & Communicati...
 
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
Cyber Infrastructure as a Service to Empower Multidisciplinary, Data-Driven S...
 
A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING
A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING
A SURVEY OF NEURAL NETWORK HARDWARE ACCELERATORS IN MACHINE LEARNING
 
Indoor 3 d video monitoring using multiple kinect depth cameras
Indoor 3 d video monitoring using multiple kinect depth camerasIndoor 3 d video monitoring using multiple kinect depth cameras
Indoor 3 d video monitoring using multiple kinect depth cameras
 
Indoor 3D Video Monitoring Using Multiple Kinect Depth-Cameras
Indoor 3D Video Monitoring Using Multiple Kinect Depth-CamerasIndoor 3D Video Monitoring Using Multiple Kinect Depth-Cameras
Indoor 3D Video Monitoring Using Multiple Kinect Depth-Cameras
 
Grid computing [2005]
Grid computing [2005]Grid computing [2005]
Grid computing [2005]
 
Network Centric to Data Centric Networks (Data Distribution Service)
Network Centric to Data Centric Networks (Data Distribution Service)Network Centric to Data Centric Networks (Data Distribution Service)
Network Centric to Data Centric Networks (Data Distribution Service)
 
team12.project_ver_1_(1).pptx
team12.project_ver_1_(1).pptxteam12.project_ver_1_(1).pptx
team12.project_ver_1_(1).pptx
 
Designing High performance & Scalable Middleware for HPC
Designing High performance & Scalable Middleware for HPCDesigning High performance & Scalable Middleware for HPC
Designing High performance & Scalable Middleware for HPC
 
Ericsson Technology Review: The future of cloud computing: Highly distributed...
Ericsson Technology Review: The future of cloud computing: Highly distributed...Ericsson Technology Review: The future of cloud computing: Highly distributed...
Ericsson Technology Review: The future of cloud computing: Highly distributed...
 
Designing High-Performance and Scalable Middleware for HPC, AI and Data Science
Designing High-Performance and Scalable Middleware for HPC, AI and Data ScienceDesigning High-Performance and Scalable Middleware for HPC, AI and Data Science
Designing High-Performance and Scalable Middleware for HPC, AI and Data Science
 
(R)evolution of the computing continuum - A few challenges
(R)evolution of the computing continuum  - A few challenges(R)evolution of the computing continuum  - A few challenges
(R)evolution of the computing continuum - A few challenges
 
Deep Learning Neural Networks in the Cloud
Deep Learning Neural Networks in the CloudDeep Learning Neural Networks in the Cloud
Deep Learning Neural Networks in the Cloud
 
Rohan resume
Rohan resumeRohan resume
Rohan resume
 
HARDWARE DESIGN FOR MACHINE LEARNING
HARDWARE DESIGN FOR MACHINE LEARNINGHARDWARE DESIGN FOR MACHINE LEARNING
HARDWARE DESIGN FOR MACHINE LEARNING
 
Hardware design for_machine_learning
Hardware design for_machine_learningHardware design for_machine_learning
Hardware design for_machine_learning
 
Grid Computing
Grid ComputingGrid Computing
Grid Computing
 
PIMRC-2012, Sydney, Australia, 28 July, 2012
PIMRC-2012, Sydney, Australia, 28 July, 2012PIMRC-2012, Sydney, Australia, 28 July, 2012
PIMRC-2012, Sydney, Australia, 28 July, 2012
 

More from OKOKPROJECTS

More from OKOKPROJECTS (20)

Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...
Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...
Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...
 
Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...
Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...
Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...
 
DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...
DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...
DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...
 
DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...
DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...
DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...
 
Digital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdf
Digital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdfDigital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdf
Digital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdf
 
Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...
Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...
Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...
 
DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...
DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...
DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...
 
Dense Nested Attention Network for Infrared Small Target Detection.pdf
Dense Nested Attention Network for Infrared Small Target Detection.pdfDense Nested Attention Network for Infrared Small Target Detection.pdf
Dense Nested Attention Network for Infrared Small Target Detection.pdf
 
Detection of Small Moving Targets in Cluttered Infrared Imagery.pdf
Detection of Small Moving Targets in Cluttered Infrared Imagery.pdfDetection of Small Moving Targets in Cluttered Infrared Imagery.pdf
Detection of Small Moving Targets in Cluttered Infrared Imagery.pdf
 
Depression Screening in Humans With AI and Deep Learning Techniques.pdf
Depression Screening in Humans With AI and Deep Learning Techniques.pdfDepression Screening in Humans With AI and Deep Learning Techniques.pdf
Depression Screening in Humans With AI and Deep Learning Techniques.pdf
 
DeepTx Deep Learning Beamforming With Channel Prediction.pdf
DeepTx Deep Learning Beamforming With Channel Prediction.pdfDeepTx Deep Learning Beamforming With Channel Prediction.pdf
DeepTx Deep Learning Beamforming With Channel Prediction.pdf
 
DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...
DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...
DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...
 
DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...
DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...
DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...
 
Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...
Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...
Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...
 
Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...
Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...
Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...
 
DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...
DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...
DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...
 
DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...
DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...
DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...
 
D2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdf
D2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdfD2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdf
D2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdf
 
Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...
Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...
Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...
 
Cyber Code Intelligence for Android Malware Detection.pdf
Cyber Code Intelligence for Android Malware Detection.pdfCyber Code Intelligence for Android Malware Detection.pdf
Cyber Code Intelligence for Android Malware Detection.pdf
 

Recently uploaded

Spellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPSSpellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPS
AnaAcapella
 

Recently uploaded (20)

The Story of Village Palampur Class 9 Free Study Material PDF
The Story of Village Palampur Class 9 Free Study Material PDFThe Story of Village Palampur Class 9 Free Study Material PDF
The Story of Village Palampur Class 9 Free Study Material PDF
 
Book Review of Run For Your Life Powerpoint
Book Review of Run For Your Life PowerpointBook Review of Run For Your Life Powerpoint
Book Review of Run For Your Life Powerpoint
 
How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17
 
Spring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community PartnershipsSpring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community Partnerships
 
Spellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPSSpellings Wk 4 and Wk 5 for Grade 4 at CAPS
Spellings Wk 4 and Wk 5 for Grade 4 at CAPS
 
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
 
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
 
Improved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio AppImproved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio App
 
PSYPACT- Practicing Over State Lines May 2024.pptx
PSYPACT- Practicing Over State Lines May 2024.pptxPSYPACT- Practicing Over State Lines May 2024.pptx
PSYPACT- Practicing Over State Lines May 2024.pptx
 
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
 
An Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge AppAn Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge App
 
How to Manage Website in Odoo 17 Studio App.pptx
How to Manage Website in Odoo 17 Studio App.pptxHow to Manage Website in Odoo 17 Studio App.pptx
How to Manage Website in Odoo 17 Studio App.pptx
 
Basic Civil Engineering notes on Transportation Engineering & Modes of Transport
Basic Civil Engineering notes on Transportation Engineering & Modes of TransportBasic Civil Engineering notes on Transportation Engineering & Modes of Transport
Basic Civil Engineering notes on Transportation Engineering & Modes of Transport
 
UChicago CMSC 23320 - The Best Commit Messages of 2024
UChicago CMSC 23320 - The Best Commit Messages of 2024UChicago CMSC 23320 - The Best Commit Messages of 2024
UChicago CMSC 23320 - The Best Commit Messages of 2024
 
AIM of Education-Teachers Training-2024.ppt
AIM of Education-Teachers Training-2024.pptAIM of Education-Teachers Training-2024.ppt
AIM of Education-Teachers Training-2024.ppt
 
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjjStl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjj
 
An overview of the various scriptures in Hinduism
An overview of the various scriptures in HinduismAn overview of the various scriptures in Hinduism
An overview of the various scriptures in Hinduism
 
Observing-Correct-Grammar-in-Making-Definitions.pptx
Observing-Correct-Grammar-in-Making-Definitions.pptxObserving-Correct-Grammar-in-Making-Definitions.pptx
Observing-Correct-Grammar-in-Making-Definitions.pptx
 
ANTI PARKISON DRUGS.pptx
ANTI         PARKISON          DRUGS.pptxANTI         PARKISON          DRUGS.pptx
ANTI PARKISON DRUGS.pptx
 
OS-operating systems- ch05 (CPU Scheduling) ...
OS-operating systems- ch05 (CPU Scheduling) ...OS-operating systems- ch05 (CPU Scheduling) ...
OS-operating systems- ch05 (CPU Scheduling) ...
 

A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices With Heterogeneous Hardware.pdf

  • 1. A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices With Heterogeneous Hardware Abstract With the emergence of more and more powerful chipsets and hardware and the rise of Artificial Intelligence of Things (AIoT bringing Deep Neural Network (DNN) models to empower mobile and edge devices with intelligence such that they can support attractive AI applications on the edge in a real heterogeneous computational resources (such as CPU, GPU, DSP, etc) to effectively and efficiently support concurrent inference of multiple DNN models on a mobile or edge device, we propose a novel online Co framework based on deep REinforcement Learning ( COSREL. COSREL has the following desirable features: 1) it achieves significant speedup over commonly computational resources on heterogeneous hardware; 2) it leverages emerging Deep Reinforcement Learning (DRL) to make dynamic and wise Scheduling Framework for DNN Models on Mobile and Edge Devices With Heterogeneous Hardware With the emergence of more and more powerful chipsets and hardware and the rise of Artificial Intelligence of Things (AIoT), there is a growing trend for bringing Deep Neural Network (DNN) models to empower mobile and edge devices with intelligence such that they can support attractive AI applications on the edge in a real-time or near real-time manner. To leverage ous computational resources (such as CPU, GPU, DSP, etc) to effectively and efficiently support concurrent inference of multiple DNN models on a mobile or edge device, we propose a novel online Co framework based on deep REinforcement Learning (DRL), which we call COSREL. COSREL has the following desirable features: 1) it achieves significant speedup over commonly-used methods by efficiently utilizing all the computational resources on heterogeneous hardware; 2) it leverages cement Learning (DRL) to make dynamic and wise Scheduling Framework for DNN Models on Mobile and Edge Devices With the emergence of more and more powerful chipsets and hardware and ), there is a growing trend for bringing Deep Neural Network (DNN) models to empower mobile and edge devices with intelligence such that they can support attractive AI applications time manner. To leverage ous computational resources (such as CPU, GPU, DSP, etc) to effectively and efficiently support concurrent inference of multiple DNN models on a mobile or edge device, we propose a novel online Co-Scheduling DRL), which we call COSREL. COSREL has the following desirable features: 1) it achieves used methods by efficiently utilizing all the computational resources on heterogeneous hardware; 2) it leverages cement Learning (DRL) to make dynamic and wise
  • 2. online scheduling decisions based on system runtime state; 3) it is capable of making a good tradeoff among inference latency, throughput and energy efficiency; and 4) it makes no changes to given DNN models, thus preserves their accuracies. To validate and evaluate COSREL, we conduct extensive experiments on an off-the-shelf Android smartphone with widely-used DNN models to compare it with three commonly-used baselines. Our experimental results show that 1) COSREL consistently and significantly outperforms all the baselines in terms of both throughput and latency; and 2) COSREL is generally superior to all the baselines in terms of energy efficiency.