SlideShare a Scribd company logo

[NeuralIPS 2020]filter in filter pruning

•Download as PPTX, PDF•
•
KIMMINHA3
KIMMINHA3

They proposed two novel methods. 1. Stripe-Wise Pruning (SWP) They propose a new pruning paradigm called SWP (Stripe-Wise Pruning) They achieve a higher pruning ratio compared to the filter-wise, channel-wise, and group-wise pruning methods. 2. Filter Skeleton (FS) They propose a new method ‘Filter Skeleton’ to efficiently learn the optimal shape of the filters for pruning. They didn't much compare with other baselines. But they obviously suggested the novel methods, that is why I choose for review when reviewing the paper. More, they said that It is State-of-the-art (SOTA) method of lately pruning methods.

Technology
1 of 18
Data-driven AI Security HCI (DASH) Lab 1 Data-driven AI Security HCI (DASH) Lab Pruning Filter in Filter 김민하 소프트웨어학과 성균관대학교 NeuralIPS 2020 May 6, 2020 Data-driven AI Security HCI (DASH) Lab
Data-driven AI Security HCI (DASH) Lab Pruning? [NIPS2015] Learning both Weights and Connections for Efficient Neural Networks https://arxiv.org/abs/1506.02626 To remove the weight in neural network model Weight Pruning Weight Pruning (WP) prunes weight of each filter It remove redundant neurons iteratively
Data-driven AI Security HCI (DASH) Lab Pruning? C : number of channels N : number of output channels W,H : Width / High Filter/Channel Pruning Filter/Channel Pruning (FP) prunes at the level of filter and channel It can prune a large region compared with the weight pruning 1. 2.
Data-driven AI Security HCI (DASH) Lab Pruning? C : number of channels N : number of output channels W,H : Width / High Group Pruning  It breaks the independent assumption on the filters  Although the position in each filter same, the importance of weights is different  The network may lose representation ability under a large pruning ratio
Data-driven AI Security HCI (DASH) Lab Abstract model deployment is sometimes costly due to the large number of parameters in DNNs  To solve this, ‘Pruning’ which is one of model compression algorithms  Filter Pruning(FP), Channel Pruning (CP), Weight Pruning (WP) and Group Pruning (GP)… Proble m Backgrou nd However, these can lose Important information because of pruning the weights of the same position  They wonder if they can learn the optimal kernel size of each filter by pruning
Data-driven AI Security HCI (DASH) Lab Abstract Solutio n To converge the strength of filter pruning and weight pruning, they propose method as Stripe-Wise Pruning (SWP) with Filter Skeleton (FS). To fine the ‘filter shape’ alongside the filter weights, they propose ‘Filter Skeleton (FS)’ It treat a filter as K x K stripes, by pruning the stripes instead of the whole filter It can achieve finer granularity than traditional FP while being hardware friendly
Data-driven AI Security HCI (DASH) Lab C : number of channels N : number of output channels W,H : Width / High Stripe Pruning It keeps each filter independent with each other, thus can lead to a more efficient network structure. (Their proposed method) Proposed Method – Filter Skeleton(FS)
Data-driven AI Security HCI (DASH) Lab Proposed Method – Filter Skeleton(FS) : # of filters : channels : Kernel size : Filter Skeleton : feature map height : feature map width Filter Skeleton (FS) • Learnable matrix that reflects the shape of each filter • Values of FS first initialized to 1 • l-th convolutional layer’s weight W : 𝑅𝑁×𝐶×𝐾×𝐾 → FS : 𝑅𝑁×𝐾×𝐾 Each filter has one FS
Data-driven AI Security HCI (DASH) Lab Proposed Method – Filter Skeleton(FS) ● Filter Skeleton(FS) ○ Loss function ○ Gradient of W(Weight), I(Filter Skeleton) ○ Gradient of W, I (1) (2) (3) (4) : # of filters : channels : Kernel size : Filter Skeleton : feature map height : feature map width
Data-driven AI Security HCI (DASH) Lab Proposed Method – Filter Skeleton(FS)  Mini-figure  one of 9 strips  X-axis  all the filters (N)  Y-axis  summation of the stripes located in the same position of all the filters Filter (3x3)
Data-driven AI Security HCI (DASH) Lab Proposed Method – Stripe-wise pruning (SWP) • Stripe-wise pruning (SWP) • Set threshold δ • Corresponding values in FS < δ → not updated during training → pruned α : magnitude of regularization g(I) : L1 norm penalty on ‘I’ (5) (6)
Data-driven AI Security HCI (DASH) Lab Proposed Method – Stripe-wise pruning (SWP) (7)
Data-driven AI Security HCI (DASH) Lab Experiments-Group Pruning vs. Stripe Pruning Can continue the training Can not continue the training Group-wise pruning (GP) They find that in GW, layer2.7 filters will be identified as invalid because all the weights are removed while training  It can not continue the training Stripe-wise pruning (SWP) Stripe-wise pruning keeps each filter independent of each other  It can continue the training and achieve a higher accuracy than GP
Data-driven AI Security HCI (DASH) Lab Experiments ● Baseline accuracy (CIFAR-10) ○ VGG16 93.25% ○ ResNet56 93.1% ● Baseline accuracy (ImageNet) ○ ResNet18 - Top-1 69.76% - Top-5 89.08%
Data-driven AI Security HCI (DASH) Lab • White color denotes the corresponding strip in the filter is removed by SWP • In the layer that close to input, most preserved layers have multiple strips • In middle layers, SWP only have one strip  redundancy is decreased Experiments - Visualization of the filters pruned by SWP (VGG19) Layer # Display the filters according to their frequency in such layer Highest Frequency Lowest Frequency
Data-driven AI Security HCI (DASH) Lab Experiments – Ablation Study • How hyper-parameters affect pruning results • Changing α (magnitude of regularization), δ (threshold) • α = 1e-5, δ = 0.05 gives the acceptable pruning ratio and test accuracy
Data-driven AI Security HCI (DASH) Lab Conclusion • Stripe-Wise Pruning (SWP) - They propose a new pruning paradigm called SWP (Stripe-Wise Pruning) - They achieve a higher pruning ratio compared to the filter-wise and group-wise pruning methods. - It achieve finer granularity than traditional FP while being hardware friendly • Filter Skeleton (FS) - They propose a new method ‘Filter Skeleton’ to efficiently learn the optimal shape of the filters for pruning - Through extensive experiments and analyses, they demonstrate the effectiveness • Achievement SOTA pruning ratio - They show SWP achieves state-of-art pruning ratio on CIFAR-10 and ImageNet datasets compared to filter-wise, channel-wise or group- wise pruning
Data-driven AI Security HCI (DASH) Lab Thank you !

Recommended

Gradient descent method
Gradient descent methodGradient descent method
Gradient descent methodProf. Neeta Awasthy
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories taeseon ryu
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Optimization in Deep Learning
Optimization in Deep LearningOptimization in Deep Learning
Optimization in Deep LearningYan Xu
 
Back propagation
Back propagationBack propagation
Back propagationNagarajan
 
Support Vector Machines- SVM
Support Vector Machines- SVMSupport Vector Machines- SVM
Support Vector Machines- SVMCarlo Carandang
 
Feedforward neural network
Feedforward neural networkFeedforward neural network
Feedforward neural networkSopheaktra YONG
 
Feature selection
Feature selectionFeature selection
Feature selectiondkpawar
 

Recommended

Gradient descent method
Gradient descent methodGradient descent method
Gradient descent methodProf. Neeta Awasthy
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories taeseon ryu
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Optimization in Deep Learning
Optimization in Deep LearningOptimization in Deep Learning
Optimization in Deep LearningYan Xu
 
Back propagation
Back propagationBack propagation
Back propagationNagarajan
 
Support Vector Machines- SVM
Support Vector Machines- SVMSupport Vector Machines- SVM
Support Vector Machines- SVMCarlo Carandang
 
Feedforward neural network
Feedforward neural networkFeedforward neural network
Feedforward neural networkSopheaktra YONG
 
Feature selection
Feature selectionFeature selection
Feature selectiondkpawar
 
Simple Introduction to AutoEncoder
Simple Introduction to AutoEncoderSimple Introduction to AutoEncoder
Simple Introduction to AutoEncoderJun Lang
 
Machine learning Lecture 1
Machine learning Lecture 1Machine learning Lecture 1
Machine learning Lecture 1Srinivasan R
 
Linear models and multiclass classification
Linear models and multiclass classificationLinear models and multiclass classification
Linear models and multiclass classificationNdSv94
 
K Nearest Neighbors
K Nearest NeighborsK Nearest Neighbors
K Nearest NeighborsTilani Gunawardena PhD(UNIBAS), BSc(Pera), FHEA(UK), CEng, MIESL
 
Kernels and Support Vector Machines
Kernels and Support Vector MachinesKernels and Support Vector Machines
Kernels and Support Vector MachinesEdgar Marca
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronMostafa G. M. Mostafa
 
Machine Learning and Data Mining: 10 Introduction to Classification
Machine Learning and Data Mining: 10 Introduction to ClassificationMachine Learning and Data Mining: 10 Introduction to Classification
Machine Learning and Data Mining: 10 Introduction to ClassificationPier Luca Lanzi
 
Machine Learning: Bias and Variance Trade-off
Machine Learning: Bias and Variance Trade-offMachine Learning: Bias and Variance Trade-off
Machine Learning: Bias and Variance Trade-offInternational Institute of Information Technology (I²IT)
 
Decision trees in Machine Learning
Decision trees in Machine Learning Decision trees in Machine Learning
Decision trees in Machine Learning Mohammad Junaid Khan
 
RM 701 Genetic Algorithm and Fuzzy Logic lecture
RM 701 Genetic Algorithm and Fuzzy Logic lectureRM 701 Genetic Algorithm and Fuzzy Logic lecture
RM 701 Genetic Algorithm and Fuzzy Logic lectureVIT University (Chennai Campus)
 
[2020 CVPR Efficient DET paper review]
[2020 CVPR Efficient DET paper review][2020 CVPR Efficient DET paper review]
[2020 CVPR Efficient DET paper review]taeseon ryu
 
Transfer Learning
Transfer LearningTransfer Learning
Transfer LearningHichem Felouat
 
Activation function
Activation functionActivation function
Activation functionAstha Jain
 
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & KamberChapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kambererror007
 
Support Vector Machines ( SVM )
Support Vector Machines ( SVM ) Support Vector Machines ( SVM )
Support Vector Machines ( SVM ) Mohammad Junaid Khan
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural NetworksDatabricks
 
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAIGenerative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAIWithTheBest
 
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...Edureka!
 
Deep Learning - RNN and CNN
Deep Learning - RNN and CNNDeep Learning - RNN and CNN
Deep Learning - RNN and CNNPradnya Saval
 
Support vector machine-SVM's
Support vector machine-SVM'sSupport vector machine-SVM's
Support vector machine-SVM'sAnudeep Chowdary Kamepalli
 
INDUSTRIAL TRAINING REPORT
INDUSTRIAL TRAINING REPORTINDUSTRIAL TRAINING REPORT
INDUSTRIAL TRAINING REPORTABHISHEK DABRAL
 
P1121106496
P1121106496P1121106496
P1121106496Ashraf Aboshosha
 

More Related Content

What's hot

Simple Introduction to AutoEncoder
Simple Introduction to AutoEncoderSimple Introduction to AutoEncoder
Simple Introduction to AutoEncoderJun Lang
 
Machine learning Lecture 1
Machine learning Lecture 1Machine learning Lecture 1
Machine learning Lecture 1Srinivasan R
 
Linear models and multiclass classification
Linear models and multiclass classificationLinear models and multiclass classification
Linear models and multiclass classificationNdSv94
 
Kernels and Support Vector Machines
Kernels and Support Vector MachinesKernels and Support Vector Machines
Kernels and Support Vector MachinesEdgar Marca
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronMostafa G. M. Mostafa
 
Machine Learning and Data Mining: 10 Introduction to Classification
Machine Learning and Data Mining: 10 Introduction to ClassificationMachine Learning and Data Mining: 10 Introduction to Classification
Machine Learning and Data Mining: 10 Introduction to ClassificationPier Luca Lanzi
 
Decision trees in Machine Learning
Decision trees in Machine Learning Decision trees in Machine Learning
Decision trees in Machine Learning Mohammad Junaid Khan
 
[2020 CVPR Efficient DET paper review]
[2020 CVPR Efficient DET paper review][2020 CVPR Efficient DET paper review]
[2020 CVPR Efficient DET paper review]taeseon ryu
 
Activation function
Activation functionActivation function
Activation functionAstha Jain
 
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & KamberChapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kambererror007
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural NetworksDatabricks
 
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAIGenerative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAIWithTheBest
 
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...Edureka!
 
Deep Learning - RNN and CNN
Deep Learning - RNN and CNNDeep Learning - RNN and CNN
Deep Learning - RNN and CNNPradnya Saval
 

What's hot (20)

Simple Introduction to AutoEncoder
Simple Introduction to AutoEncoderSimple Introduction to AutoEncoder
Simple Introduction to AutoEncoder
 
Machine learning Lecture 1
Machine learning Lecture 1Machine learning Lecture 1
Machine learning Lecture 1
 
Linear models and multiclass classification
Linear models and multiclass classificationLinear models and multiclass classification
Linear models and multiclass classification
 
K Nearest Neighbors
K Nearest NeighborsK Nearest Neighbors
K Nearest Neighbors
 
Kernels and Support Vector Machines
Kernels and Support Vector MachinesKernels and Support Vector Machines
Kernels and Support Vector Machines
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer Perceptron
 
Machine Learning and Data Mining: 10 Introduction to Classification
Machine Learning and Data Mining: 10 Introduction to ClassificationMachine Learning and Data Mining: 10 Introduction to Classification
Machine Learning and Data Mining: 10 Introduction to Classification
 
Machine Learning: Bias and Variance Trade-off
Machine Learning: Bias and Variance Trade-offMachine Learning: Bias and Variance Trade-off
Machine Learning: Bias and Variance Trade-off
 
Decision trees in Machine Learning
Decision trees in Machine Learning Decision trees in Machine Learning
Decision trees in Machine Learning
 
RM 701 Genetic Algorithm and Fuzzy Logic lecture
RM 701 Genetic Algorithm and Fuzzy Logic lectureRM 701 Genetic Algorithm and Fuzzy Logic lecture
RM 701 Genetic Algorithm and Fuzzy Logic lecture
 
[2020 CVPR Efficient DET paper review]
[2020 CVPR Efficient DET paper review][2020 CVPR Efficient DET paper review]
[2020 CVPR Efficient DET paper review]
 
Transfer Learning
Transfer LearningTransfer Learning
Transfer Learning
 
Activation function
Activation functionActivation function
Activation function
 
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & KamberChapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber
Chapter - 6 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber
 
Support Vector Machines ( SVM )
Support Vector Machines ( SVM ) Support Vector Machines ( SVM )
Support Vector Machines ( SVM )
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural Networks
 
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAIGenerative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI
Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI
 
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
Deep Learning Tutorial | Deep Learning Tutorial for Beginners | Neural Networ...
 
Deep Learning - RNN and CNN
Deep Learning - RNN and CNNDeep Learning - RNN and CNN
Deep Learning - RNN and CNN
 
Support vector machine-SVM's
Support vector machine-SVM'sSupport vector machine-SVM's
Support vector machine-SVM's
 

Similar to [NeuralIPS 2020]filter in filter pruning

INDUSTRIAL TRAINING REPORT
INDUSTRIAL TRAINING REPORTINDUSTRIAL TRAINING REPORT
INDUSTRIAL TRAINING REPORTABHISHEK DABRAL
 
Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...
Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...
Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...IRJET Journal
 
Simulation of EMI Filters Using Matlab
Simulation of EMI Filters Using MatlabSimulation of EMI Filters Using Matlab
Simulation of EMI Filters Using Matlabinventionjournals
 
Circuit Theory 2: Filters Project Report
Circuit Theory 2: Filters Project ReportCircuit Theory 2: Filters Project Report
Circuit Theory 2: Filters Project ReportMichael Sandy
 
Design of Filter Circuits using MATLAB, Multisim, and Excel
Design of Filter Circuits using MATLAB, Multisim, and ExcelDesign of Filter Circuits using MATLAB, Multisim, and Excel
Design of Filter Circuits using MATLAB, Multisim, and ExcelDavid Sandy
 
International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)IJERD Editor
 
My Dissertation Presentation Slides
My Dissertation Presentation SlidesMy Dissertation Presentation Slides
My Dissertation Presentation SlidesClyde Lettsome
 
Switchable Deep Beamformer for Ultrasound Imaging Using ADAIN
Switchable Deep Beamformer for Ultrasound Imaging Using ADAINSwitchable Deep Beamformer for Ultrasound Imaging Using ADAIN
Switchable Deep Beamformer for Ultrasound Imaging Using ADAINShujaat Khan
 
Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...
Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...
Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...IJERA Editor
 
http://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdf
http://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdfhttp://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdf
http://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdfHiroshi Ono
 
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John Melonakos
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John MelonakosPT-4054, "OpenCL™ Accelerated Compute Libraries" by John Melonakos
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John MelonakosAMD Developer Central
 
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...Ian Foster
 
BC-Cancer ChimeraScan Presentation
BC-Cancer ChimeraScan PresentationBC-Cancer ChimeraScan Presentation
BC-Cancer ChimeraScan PresentationElijah Willie
 
Types Of Window Being Used For The Selected Granule
Types Of Window Being Used For The Selected GranuleTypes Of Window Being Used For The Selected Granule
Types Of Window Being Used For The Selected GranuleLeslie Lee
 
Test vector compression in Digital Testing
Test vector compression in Digital Testing Test vector compression in Digital Testing
Test vector compression in Digital Testing Amr Abd El Latief
 
A channel allocation algorithm for reducing the channel sensing reserving asy...
A channel allocation algorithm for reducing the channel sensing reserving asy...A channel allocation algorithm for reducing the channel sensing reserving asy...
A channel allocation algorithm for reducing the channel sensing reserving asy...ieeeprojectschennai
 

Similar to [NeuralIPS 2020]filter in filter pruning (20)

INDUSTRIAL TRAINING REPORT
INDUSTRIAL TRAINING REPORTINDUSTRIAL TRAINING REPORT
INDUSTRIAL TRAINING REPORT
 
P1121106496
P1121106496P1121106496
P1121106496
 
Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...
Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...
Design Band Pass FIR Digital Filter for Cut off Frequency Calculation Using A...
 
Simulation of EMI Filters Using Matlab
Simulation of EMI Filters Using MatlabSimulation of EMI Filters Using Matlab
Simulation of EMI Filters Using Matlab
 
Circuit Theory 2: Filters Project Report
Circuit Theory 2: Filters Project ReportCircuit Theory 2: Filters Project Report
Circuit Theory 2: Filters Project Report
 
Design of Filter Circuits using MATLAB, Multisim, and Excel
Design of Filter Circuits using MATLAB, Multisim, and ExcelDesign of Filter Circuits using MATLAB, Multisim, and Excel
Design of Filter Circuits using MATLAB, Multisim, and Excel
 
Noise Immune and Area Optimized Serial Interface for FPGA based Industrial In...
Noise Immune and Area Optimized Serial Interface for FPGA based Industrial In...Noise Immune and Area Optimized Serial Interface for FPGA based Industrial In...
Noise Immune and Area Optimized Serial Interface for FPGA based Industrial In...
 
International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)
 
My Dissertation Presentation Slides
My Dissertation Presentation SlidesMy Dissertation Presentation Slides
My Dissertation Presentation Slides
 
Packet sniffing
Packet sniffingPacket sniffing
Packet sniffing
 
Switchable Deep Beamformer for Ultrasound Imaging Using ADAIN
Switchable Deep Beamformer for Ultrasound Imaging Using ADAINSwitchable Deep Beamformer for Ultrasound Imaging Using ADAIN
Switchable Deep Beamformer for Ultrasound Imaging Using ADAIN
 
Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...
Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...
Design and Implementation of Digital Chebyshev Type II Filter using XSG for N...
 
http://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdf
http://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdfhttp://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdf
http://www.logos.ic.i.u-tokyo.ac.jp/~kay/papers/ccgrid2008_stable_broadcast.pdf
 
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John Melonakos
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John MelonakosPT-4054, "OpenCL™ Accelerated Compute Libraries" by John Melonakos
PT-4054, "OpenCL™ Accelerated Compute Libraries" by John Melonakos
 
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...
Computing Just What You Need: Online Data Analysis and Reduction at Extreme ...
 
BC-Cancer ChimeraScan Presentation
BC-Cancer ChimeraScan PresentationBC-Cancer ChimeraScan Presentation
BC-Cancer ChimeraScan Presentation
 
Types Of Window Being Used For The Selected Granule
Types Of Window Being Used For The Selected GranuleTypes Of Window Being Used For The Selected Granule
Types Of Window Being Used For The Selected Granule
 
Test vector compression in Digital Testing
Test vector compression in Digital Testing Test vector compression in Digital Testing
Test vector compression in Digital Testing
 
A channel allocation algorithm for reducing the channel sensing reserving asy...
A channel allocation algorithm for reducing the channel sensing reserving asy...A channel allocation algorithm for reducing the channel sensing reserving asy...
A channel allocation algorithm for reducing the channel sensing reserving asy...
 
Test vector compression
Test vector compressionTest vector compression
Test vector compression
 

More from KIMMINHA3

[ECCV2022] Generative Domain Adaptation for Face Anti-Spoofing
[ECCV2022] Generative Domain Adaptation for Face Anti-Spoofing[ECCV2022] Generative Domain Adaptation for Face Anti-Spoofing
[ECCV2022] Generative Domain Adaptation for Face Anti-SpoofingKIMMINHA3
 
[AAAI21] Self-Domain Adaptation for Face Anti-Spoofing
[AAAI21] Self-Domain Adaptation for Face Anti-Spoofing[AAAI21] Self-Domain Adaptation for Face Anti-Spoofing
[AAAI21] Self-Domain Adaptation for Face Anti-SpoofingKIMMINHA3
 
[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...
[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...
[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...KIMMINHA3
 
[TIFS'22] Learning Meta Pattern for Face Anti-Spoofing
[TIFS'22] Learning Meta Pattern for Face Anti-Spoofing[TIFS'22] Learning Meta Pattern for Face Anti-Spoofing
[TIFS'22] Learning Meta Pattern for Face Anti-SpoofingKIMMINHA3
 
[AAAI'23]Learning Polysemantic Spoof Trace
[AAAI'23]Learning Polysemantic Spoof Trace[AAAI'23]Learning Polysemantic Spoof Trace
[AAAI'23]Learning Polysemantic Spoof TraceKIMMINHA3
 
Architectures of Super-resolution (AI)
Architectures of Super-resolution (AI)Architectures of Super-resolution (AI)
Architectures of Super-resolution (AI)KIMMINHA3
 
[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...
[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...
[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...KIMMINHA3
 
Methods for interpreting and understanding deep neural networks
Methods for interpreting and understanding deep neural networksMethods for interpreting and understanding deep neural networks
Methods for interpreting and understanding deep neural networksKIMMINHA3
 
Meta learned Confidence for Few-shot Learning
Meta learned Confidence for Few-shot LearningMeta learned Confidence for Few-shot Learning
Meta learned Confidence for Few-shot LearningKIMMINHA3
 
“zero-shot” super-resolution using deep internal learning [CVPR2018]
“zero-shot” super-resolution using deep internal learning [CVPR2018]“zero-shot” super-resolution using deep internal learning [CVPR2018]
“zero-shot” super-resolution using deep internal learning [CVPR2018]KIMMINHA3
 
[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...
[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...
[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...KIMMINHA3
 
Transferable GAN-generated Images Detection Framework.
Transferable GAN-generated Images Detection Framework.Transferable GAN-generated Images Detection Framework.
Transferable GAN-generated Images Detection Framework.KIMMINHA3
 
[Seminar arxiv]fake face detection via adaptive residuals extraction network
[Seminar arxiv]fake face detection via adaptive residuals extraction network [Seminar arxiv]fake face detection via adaptive residuals extraction network
[Seminar arxiv]fake face detection via adaptive residuals extraction network KIMMINHA3
 
Xception mhkim
Xception mhkimXception mhkim
Xception mhkimKIMMINHA3
 
short text large effect measuring the impact of user reviews on android app s...
short text large effect measuring the impact of user reviews on android app s...short text large effect measuring the impact of user reviews on android app s...
short text large effect measuring the impact of user reviews on android app s...KIMMINHA3
 

More from KIMMINHA3 (15)

[ECCV2022] Generative Domain Adaptation for Face Anti-Spoofing
[ECCV2022] Generative Domain Adaptation for Face Anti-Spoofing[ECCV2022] Generative Domain Adaptation for Face Anti-Spoofing
[ECCV2022] Generative Domain Adaptation for Face Anti-Spoofing
 
[AAAI21] Self-Domain Adaptation for Face Anti-Spoofing
[AAAI21] Self-Domain Adaptation for Face Anti-Spoofing[AAAI21] Self-Domain Adaptation for Face Anti-Spoofing
[AAAI21] Self-Domain Adaptation for Face Anti-Spoofing
 
[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...
[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...
[CVPR'22] Domain Generalization via Shuffled Style Assembly for Face Anti-Spo...
 
[TIFS'22] Learning Meta Pattern for Face Anti-Spoofing
[TIFS'22] Learning Meta Pattern for Face Anti-Spoofing[TIFS'22] Learning Meta Pattern for Face Anti-Spoofing
[TIFS'22] Learning Meta Pattern for Face Anti-Spoofing
 
[AAAI'23]Learning Polysemantic Spoof Trace
[AAAI'23]Learning Polysemantic Spoof Trace[AAAI'23]Learning Polysemantic Spoof Trace
[AAAI'23]Learning Polysemantic Spoof Trace
 
Architectures of Super-resolution (AI)
Architectures of Super-resolution (AI)Architectures of Super-resolution (AI)
Architectures of Super-resolution (AI)
 
[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...
[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...
[CVPRW2021]FReTAL: Generalizing Deepfake detection using Knowledge Distillati...
 
Methods for interpreting and understanding deep neural networks
Methods for interpreting and understanding deep neural networksMethods for interpreting and understanding deep neural networks
Methods for interpreting and understanding deep neural networks
 
Meta learned Confidence for Few-shot Learning
Meta learned Confidence for Few-shot LearningMeta learned Confidence for Few-shot Learning
Meta learned Confidence for Few-shot Learning
 
“zero-shot” super-resolution using deep internal learning [CVPR2018]
“zero-shot” super-resolution using deep internal learning [CVPR2018]“zero-shot” super-resolution using deep internal learning [CVPR2018]
“zero-shot” super-resolution using deep internal learning [CVPR2018]
 
[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...
[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...
[CVPRW 2020]Real world Super-Resolution via Kernel Estimation and Noise Injec...
 
Transferable GAN-generated Images Detection Framework.
Transferable GAN-generated Images Detection Framework.Transferable GAN-generated Images Detection Framework.
Transferable GAN-generated Images Detection Framework.
 
[Seminar arxiv]fake face detection via adaptive residuals extraction network
[Seminar arxiv]fake face detection via adaptive residuals extraction network [Seminar arxiv]fake face detection via adaptive residuals extraction network
[Seminar arxiv]fake face detection via adaptive residuals extraction network
 
Xception mhkim
Xception mhkimXception mhkim
Xception mhkim
 
short text large effect measuring the impact of user reviews on android app s...
short text large effect measuring the impact of user reviews on android app s...short text large effect measuring the impact of user reviews on android app s...
short text large effect measuring the impact of user reviews on android app s...
 

Recently uploaded

Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CVKhem
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 

Recently uploaded (20)

Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 

[NeuralIPS 2020]filter in filter pruning

  • 1. Data-driven AI Security HCI (DASH) Lab 1 Data-driven AI Security HCI (DASH) Lab Pruning Filter in Filter 김민하 소프트웨어학과 성균관대학교 NeuralIPS 2020 May 6, 2020 Data-driven AI Security HCI (DASH) Lab
  • 2. Data-driven AI Security HCI (DASH) Lab Pruning? [NIPS2015] Learning both Weights and Connections for Efficient Neural Networks https://arxiv.org/abs/1506.02626 To remove the weight in neural network model Weight Pruning Weight Pruning (WP) prunes weight of each filter It remove redundant neurons iteratively
  • 3. Data-driven AI Security HCI (DASH) Lab Pruning? C : number of channels N : number of output channels W,H : Width / High Filter/Channel Pruning Filter/Channel Pruning (FP) prunes at the level of filter and channel It can prune a large region compared with the weight pruning 1. 2.
  • 4. Data-driven AI Security HCI (DASH) Lab Pruning? C : number of channels N : number of output channels W,H : Width / High Group Pruning  It breaks the independent assumption on the filters  Although the position in each filter same, the importance of weights is different  The network may lose representation ability under a large pruning ratio
  • 5. Data-driven AI Security HCI (DASH) Lab Abstract model deployment is sometimes costly due to the large number of parameters in DNNs  To solve this, ‘Pruning’ which is one of model compression algorithms  Filter Pruning(FP), Channel Pruning (CP), Weight Pruning (WP) and Group Pruning (GP)… Proble m Backgrou nd However, these can lose Important information because of pruning the weights of the same position  They wonder if they can learn the optimal kernel size of each filter by pruning
  • 6. Data-driven AI Security HCI (DASH) Lab Abstract Solutio n To converge the strength of filter pruning and weight pruning, they propose method as Stripe-Wise Pruning (SWP) with Filter Skeleton (FS). To fine the ‘filter shape’ alongside the filter weights, they propose ‘Filter Skeleton (FS)’ It treat a filter as K x K stripes, by pruning the stripes instead of the whole filter It can achieve finer granularity than traditional FP while being hardware friendly
  • 7. Data-driven AI Security HCI (DASH) Lab C : number of channels N : number of output channels W,H : Width / High Stripe Pruning It keeps each filter independent with each other, thus can lead to a more efficient network structure. (Their proposed method) Proposed Method – Filter Skeleton(FS)
  • 8. Data-driven AI Security HCI (DASH) Lab Proposed Method – Filter Skeleton(FS) : # of filters : channels : Kernel size : Filter Skeleton : feature map height : feature map width Filter Skeleton (FS) • Learnable matrix that reflects the shape of each filter • Values of FS first initialized to 1 • l-th convolutional layer’s weight W : 𝑅𝑁×𝐶×𝐾×𝐾 → FS : 𝑅𝑁×𝐾×𝐾 Each filter has one FS
  • 9. Data-driven AI Security HCI (DASH) Lab Proposed Method – Filter Skeleton(FS) ● Filter Skeleton(FS) ○ Loss function ○ Gradient of W(Weight), I(Filter Skeleton) ○ Gradient of W, I (1) (2) (3) (4) : # of filters : channels : Kernel size : Filter Skeleton : feature map height : feature map width
  • 10. Data-driven AI Security HCI (DASH) Lab Proposed Method – Filter Skeleton(FS)  Mini-figure  one of 9 strips  X-axis  all the filters (N)  Y-axis  summation of the stripes located in the same position of all the filters Filter (3x3)
  • 11. Data-driven AI Security HCI (DASH) Lab Proposed Method – Stripe-wise pruning (SWP) • Stripe-wise pruning (SWP) • Set threshold δ • Corresponding values in FS < δ → not updated during training → pruned α : magnitude of regularization g(I) : L1 norm penalty on ‘I’ (5) (6)
  • 12. Data-driven AI Security HCI (DASH) Lab Proposed Method – Stripe-wise pruning (SWP) (7)
  • 13. Data-driven AI Security HCI (DASH) Lab Experiments-Group Pruning vs. Stripe Pruning Can continue the training Can not continue the training Group-wise pruning (GP) They find that in GW, layer2.7 filters will be identified as invalid because all the weights are removed while training  It can not continue the training Stripe-wise pruning (SWP) Stripe-wise pruning keeps each filter independent of each other  It can continue the training and achieve a higher accuracy than GP
  • 14. Data-driven AI Security HCI (DASH) Lab Experiments ● Baseline accuracy (CIFAR-10) ○ VGG16 93.25% ○ ResNet56 93.1% ● Baseline accuracy (ImageNet) ○ ResNet18 - Top-1 69.76% - Top-5 89.08%
  • 15. Data-driven AI Security HCI (DASH) Lab • White color denotes the corresponding strip in the filter is removed by SWP • In the layer that close to input, most preserved layers have multiple strips • In middle layers, SWP only have one strip  redundancy is decreased Experiments - Visualization of the filters pruned by SWP (VGG19) Layer # Display the filters according to their frequency in such layer Highest Frequency Lowest Frequency
  • 16. Data-driven AI Security HCI (DASH) Lab Experiments – Ablation Study • How hyper-parameters affect pruning results • Changing α (magnitude of regularization), δ (threshold) • α = 1e-5, δ = 0.05 gives the acceptable pruning ratio and test accuracy
  • 17. Data-driven AI Security HCI (DASH) Lab Conclusion • Stripe-Wise Pruning (SWP) - They propose a new pruning paradigm called SWP (Stripe-Wise Pruning) - They achieve a higher pruning ratio compared to the filter-wise and group-wise pruning methods. - It achieve finer granularity than traditional FP while being hardware friendly • Filter Skeleton (FS) - They propose a new method ‘Filter Skeleton’ to efficiently learn the optimal shape of the filters for pruning - Through extensive experiments and analyses, they demonstrate the effectiveness • Achievement SOTA pruning ratio - They show SWP achieves state-of-art pruning ratio on CIFAR-10 and ImageNet datasets compared to filter-wise, channel-wise or group- wise pruning
  • 18. Data-driven AI Security HCI (DASH) Lab Thank you !