deepswarm optimising convolutional neural networks using swarm intelligence (ACO) amirsh.nll@gmail.com seaman university - Msc. Artificial intelligence

1. Amir Shokri
DeepSwarm uses Ant Colony Optimization (ACO)
Amirsh.nll@gmail.com
Dr. Kourosh Kiani

2. ‫چکیده‬
2
‫تمرین‬ ‫این‬ ‫در‬DeepSwarm،‫که‬‫عصبی‬ ‫معماری‬ ‫در‬ ‫جستجو‬ ‫جدید‬ ‫روش‬ ‫یک‬NAS‫که‬ ‫است‬‫هوش‬ ‫اصول‬ ‫بر‬ ‫مبتنی‬Swarm‫را‬ ‫باشد‬ ‫می‬
‫کنیم‬ ‫می‬ ‫بررسی‬.
DeepSwarm‫از‬ ‫خود‬ ‫اصلی‬ ‫هسته‬ ‫در‬Ant colony optimization‫بهترین‬ ‫جمعی‬ ‫طور‬ ‫به‬ ‫تا‬ ‫کند‬ ‫می‬ ‫استفاده‬NAS‫کند‬ ‫جستجو‬ ‫را‬.
‫پیشرفته‬ ‫جستجوی‬ ، ‫کنیم‬ ‫موثرتر‬ ‫هرچه‬ ‫را‬ ‫خود‬ ‫روش‬ ‫هرچه‬ ‫اینکه‬ ‫برای‬ ، ‫این‬ ‫از‬ ‫مهمتر‬NAS‫کنیم‬ ‫می‬ ‫ترکیب‬ ‫وزن‬ ‫از‬ ‫مجدد‬ ‫استفاده‬ ‫قابلیت‬ ‫با‬ ‫را‬.
‫ماهیت‬ ‫به‬ ‫توجه‬ ‫با‬ ، ‫این‬ ‫بر‬ ‫عالوه‬ACO،‫ببخ‬ ‫سرعت‬ ‫را‬ ‫جستجو‬ ‫فرایند‬ ‫بیشتر‬ ‫تواند‬ ‫می‬ ‫که‬ ‫باشد‬ ‫اکتشافی‬ ‫اطالعات‬ ‫دارای‬ ‫تواند‬ ‫می‬ ‫ما‬ ‫روش‬‫شد‬.

3. ‫مقدمه‬
3
‫دستی‬ ‫طراحی‬ ‫برای‬ ‫انسانی‬ ‫مهندسان‬ ، ‫اخیر‬ ‫سالهای‬ ‫در‬Deep NAS‫خاص‬ ‫کارهای‬ ‫برای‬‫اند‬ ‫کرده‬ ‫تالش‬.
‫است‬ ‫زیر‬ ‫واقعیت‬ ‫دو‬ ‫دلیل‬ ‫به‬ ‫عمدتا‬ ‫این‬:
(1)‫پارامترهای‬ ‫و‬ ‫ها‬ ‫الیه‬ ‫با‬ ‫مدرن‬ ‫عمیق‬ ‫عصبی‬ ‫معماری‬‫زیاد‬‫است‬ ‫پیچیده‬ ‫بسیار‬.
(2)‫باشد‬ ‫ضعیف‬ ‫دیگران‬ ‫مورد‬ ‫در‬ ‫اما‬ ‫باشد‬ ‫داشته‬ ‫مشکالت‬ ‫از‬ ‫نوع‬ ‫یک‬ ‫در‬ ‫یا‬ ‫داده‬ ‫مجموعه‬ ‫یک‬ ‫در‬ ‫خوبی‬ ‫عملکرد‬ ‫است‬ ‫ممکن‬ ‫معماری‬ ‫یک‬.
‫این‬‫طراحی‬ ‫بتواند‬ ‫که‬ ‫دارد‬ ‫روشهایی‬ ‫ایجاد‬ ‫در‬ ‫سعی‬ ‫که‬ ‫است‬ ‫شده‬ ‫تحقیقات‬ ‫رونق‬ ‫به‬ ‫منجر‬ ‫عامل‬ ‫دو‬‫های‬ ‫معماری‬‫اتوماسیون‬ ‫را‬ ‫عصبی‬‫کن‬‫د‬.

4. ‫مقدمه‬
4
‫این‬ ‫در‬‫تمرین‬‫هوش‬ ‫بر‬ ‫مبتنی‬ ‫جدید‬ ‫عصبی‬ ‫معماری‬ ‫جستجوی‬ ‫روش‬ ‫یک‬Swarm (SI)‫ارائه‬‫است‬ ‫شده‬.
‫عصبی‬ ‫های‬ ‫شبکه‬ ‫روی‬ ‫بر‬ ‫ما‬ ، ‫شروع‬ ‫برای‬Convolutional (CNN)،‫کنیم‬ ‫می‬ ‫تمرکز‬ ، ‫عمیق‬ ‫عصبی‬ ‫های‬ ‫معماری‬ ‫ترین‬ ‫معمول‬ ‫از‬ ‫یکی‬.
‫جدید‬ ‫های‬ ‫معماری‬ ‫کشف‬ ‫برای‬CNN،‫سازی‬ ‫بهینه‬ ‫از‬ ‫ما‬ ‫روش‬AntColony (ACO)‫کند‬ ‫می‬ ‫استفاده‬
‫انگیزه‬‫از‬ ‫استفاده‬SI‫برای‬NAS‫که‬ ‫است‬ ‫دلیل‬ ‫این‬ ‫به‬SI‫مشکالت‬ ‫با‬ ‫برخورد‬ ‫هنگام‬ ‫تواند‬ ‫می‬ ‫که‬ ‫است‬ ‫جذاب‬ ‫های‬ ‫ویژگی‬ ‫از‬ ‫بسیاری‬ ‫دارای‬NAS
‫باشد‬ ‫کننده‬ ‫کمک‬.
‫است‬ ‫مورد‬ ‫چند‬ ‫ذکر‬ ‫برای‬ ‫فقط‬ ، ‫دانش‬ ‫ترکیب‬ ‫و‬ ‫اشتراک‬ ‫توانایی‬ ‫و‬ ‫پذیری‬ ‫مقیاس‬ ، ‫تمرکز‬ ‫عدم‬ ، ‫خطا‬ ‫تحمل‬ ‫شامل‬ ‫این‬.، ‫خاص‬ ‫طور‬ ‫به‬ACO
‫حوزه‬ ‫در‬ ‫طبیعی‬ ‫طور‬ ‫به‬ ‫را‬ ‫آن‬ ‫که‬ ‫دارد‬ ‫کمی‬ ‫مشخص‬ ‫های‬ ‫ویژگی‬NAS‫نمی‬ ‫قرار‬‫دهد‬.

5. ‫مقدمه‬
5
ACO‫پویا‬ ‫محیط‬ ‫با‬ ‫راحتی‬ ‫به‬ ‫تواند‬ ‫می‬ ‫و‬ ‫است‬ ‫خوب‬ ‫شود‬ ‫داده‬ ‫نشان‬ ‫نمودار‬ ‫عنوان‬ ‫به‬ ‫تواند‬ ‫می‬ ‫که‬ ‫ای‬ ‫گسسته‬ ‫مشکالت‬ ‫حل‬ ‫در‬(‫تغیی‬‫نمودار‬ ‫ر‬)
‫باشد‬ ‫سازگار‬.
‫یکی‬‫از‬ ‫استفاده‬ ‫برای‬ ‫مهم‬ ‫انگیزه‬ ‫ایجاد‬ ‫عوامل‬ ‫از‬ ‫دیگر‬SI‫متن‬ ‫در‬ ‫آن‬ ‫روشهای‬ ‫اکثر‬ ‫که‬ ‫است‬ ‫واقعیت‬ ‫این‬NAS‫است‬ ‫نگرفته‬ ‫قرار‬ ‫بررسی‬ ‫مورد‬.

6. DeepSwarm
6
𝑠 =
arg max{ 𝑡 𝑟, 𝑢 . 𝜂 𝑟, 𝑢 𝛽, 𝑖𝑓 𝑞 ≤ 𝑞0
𝑢 ∈ 𝐽 𝑘(𝑟)
𝑆, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝑝 𝑘 𝑟, 𝑠 =
𝑡 𝑟, 𝑠 . 𝜂 𝑟, 𝑠 𝛽
Σ[𝑡 𝑟, 𝑢 . 𝜂 𝑟, 𝑢 𝛽
𝑢 ∈ 𝐽 𝑘(𝑟)
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝑖𝑓 𝑠 ∈ 𝑗 𝑘(𝑟)

7. DeepSwarm
7
τ 𝑟, 𝑠 < − 1 − 𝜌 . 𝜏 𝑟, 𝑠 + 𝜌. 𝜏0
Δ𝜏 𝑟, 𝑠 =
𝐶𝑔𝑏 𝑖𝑓 𝑟, 𝑠 ∈ 𝑔𝑙𝑜𝑏𝑎𝑙 − 𝑏𝑒𝑠𝑡 − 𝑡𝑜𝑢𝑟
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

8. ‫روند‬NAS
8
‫روند‬NAS‫در‬DEEPSwarm‫الگوریتم‬ ‫با‬ACS

9. ‫روش‬‫ارزیابی‬-‫الگوریتم‬DeepSwarm–‫تابع‬search
9
Function search():
graph = Graph
while graph.current depth < max depth do
ants = generate ants()
best ant = find best(ants)
graph.global pheromone update(best ant)
graph.increase depth()
return best ant

10. ‫روش‬‫ارزیابی‬-‫الگوریتم‬DeepSwarm–‫تابع‬generateants
10
Function generate ants():
ants = []
for i = 0 to ant count do
ant = Ant()
ant.path = generate path()
ant.evaluate()
ants.append(ant)
graph.local pheromone update(ant)
return ants

11. ‫روش‬‫ارزیابی‬-‫الگوریتم‬DeepSwarm–‫تابع‬generatepath
11
Function generate path():
current node = graph.input
node path = [current node]
for i = 0 to current max depth do
if current node.neighbours ←−∅ then
break
current node = aco select rule(current node.neighbours)
path.append(current node)
completed path = complete path(path)
return path

12. ‫روش‬‫ارزیابی‬-‫الگوریتم‬DeepSwarm–‫تابع‬acoselect
12
Function aco select(neighbours):
foreach neighbour ∈ neighbours do
probability = neighbour.pheromone × neighbour.heuristic
probabilities ← probability
denominator += probability
if random.uniform(0, 1) ≤ greediness then
max index = probabilities.index(max(probabilities))
return neighbours[max index]
probabilities = probabilities / denominator
neighbour index = wheel selection(probabilities)
return neighbours[neighbour index]

13. AntCount
13
‫دیتاست‬ ‫در‬ ‫خطا‬ ‫میزان‬CIFAR-10‫از‬ ‫قبل‬training‫نهایی‬(‫پنج‬trials‫مجزا‬)

14. AntCount
14
‫دیتاست‬ ‫در‬ ‫خطا‬ ‫میزان‬CIFAR-10‫از‬ ‫بعد‬training‫نهایی‬(‫پنج‬trials‫مجزا‬)

15. AntCount
15
‫اجرا‬ ‫زمان‬ ‫میانگین‬(‫پنج‬trials‫مختلف‬)‫ساعت‬ ‫در‬–ant‫دیتاست‬ ‫در‬ ‫مختلف‬ ‫های‬CIFAR-10

16. Greediness
16
‫دیتاست‬ ‫در‬ ‫خطا‬ ‫میزان‬CIFAR-10‫از‬ ‫قبل‬training‫نهایی‬(‫پنج‬trials‫مجزا‬)

17. Greediness
17
‫دیتاست‬ ‫در‬ ‫خطا‬ ‫میزان‬CIFAR-10‫از‬ ‫بعد‬training‫نهایی‬(‫پنج‬trials‫مجزا‬)

18. ‫یا‬‫صحت‬Accuracy
18
method MNIST Fashion-MNIST CIFAR-10
Random 1.79% 11.36% 16.86%
Grid 1.68% 10.28% 17.17%
SPMT 1.36% 9.62% 14.68%
SMAC 1.43% 10.87% 15.04%
SEAS 1.07% 9.05% 12.43%
NASBOT N/A N/A 12.30%
AutoKeras BFS 1.56% 9.13% 13.84%
AutoKeras BO 1.83% 7.99% 12.90%
AutoKeras BFS 0.55% 7.42% 11.44%
DeepSwarm Average 0.46% 6.75% 12.70%
DeepSwarm Best 0.39% 6.44% 11.31%
‫دیتاست‬ ‫در‬ ‫خطا‬ ‫میزان‬CIFAR-10

19. Appendix
19
‫در‬ ‫موجود‬ ‫های‬ ‫معماری‬ ‫بهترین‬DeepSwarm

20. ‫پایتون‬‫کد‬–cifar10.py
20
import context
import tensorflow as tf
from deepswarm.backends import Dataset,TFKerasBackend
from deepswarm.deepswarm import DeepSwarm
cifar10 = tf.keras.datasets.cifar10
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
y_train = tf.keras.utils.to_categorical(y_train, 10)
y_test = tf.keras.utils.to_categorical(y_test, 10)
dataset = Dataset(
training_examples=x_train,
training_labels=y_train,
testing_examples=x_test,
testing_labels=y_test,
validation_split=0.1,
)
backend =TFKerasBackend(dataset=dataset)
deepswarm = DeepSwarm(backend=backend)
topology = deepswarm.find_topology()

21. ‫پایتون‬‫کد‬–cifar10.py
21
deepswarm.evaluate_topology(topology)
trained_topology = deepswarm.train_topology(topology, 50, augment={
'rotation_range': 15,
'width_shift_range': 0.1,
'height_shift_range': 0.1,
'horizontal_flip':True,
})
deepswarm.evaluate_topology(trained_topology)

22. ‫پایتون‬‫کد‬–context.py
22
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))

23. ‫پایتون‬‫کد‬–mnist.py
23
import context
import tensorflow as tf
from deepswarm.backends import Dataset,TFKerasBackend
from deepswarm.deepswarm import DeepSwarm
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
x_train = x_train.reshape(x_train.shape[0], 28, 28, 1)
x_test = x_test.reshape(x_test.shape[0], 28, 28, 1)
normalized_dataset = Dataset(
training_examples=x_train,
training_labels=y_train,
testing_examples=x_test,
testing_labels=y_test,
validation_split=0.1,
)
backend =TFKerasBackend(dataset=normalized_dataset)
deepswarm = DeepSwarm(backend=backend)

24. ‫پایتون‬‫کد‬–mnist.py
24
topology = deepswarm.find_topology()
deepswarm.evaluate_topology(topology)
trained_topology = deepswarm.train_topology(topology, 30)
deepswarm.evaluate_topology(trained_topology)

25. ‫پایتون‬‫کد‬–fashiop-mnist.py
25
import context
import tensorflow as tf
from deepswarm.backends import Dataset,TFKerasBackend
from deepswarm.deepswarm import DeepSwarm
fashion_mnist = tf.keras.datasets.fashion_mnist
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
x_train = x_train.reshape(x_train.shape[0], 28, 28, 1)
x_test = x_test.reshape(x_test.shape[0], 28, 28, 1)
normalized_dataset = Dataset(
training_examples=x_train,
training_labels=y_train,
testing_examples=x_test,
testing_labels=y_test,
validation_split=0.1,
)

26. ‫پایتون‬‫کد‬–fashiop-mnist.py
26
backend =TFKerasBackend(dataset=normalized_dataset)
deepswarm = DeepSwarm(backend=backend)
topology = deepswarm.find_topology()
deepswarm.evaluate_topology(topology)
trained_topology = deepswarm.train_topology(topology, 50)
deepswarm.evaluate_topology(trained_topology)

27. ‫منابع‬
27
• https://arxiv.org/pdf/1905.07350v1.pdf
• https://github.com/Pattio/DeepSwarm
• 1. Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 (2016)
• 2. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. Journal of Machine Learning Research 13(Feb), 281–305 (2012)
• 3. Desell, T., Clachar, S., Higgins, J., Wild, B.: Evolving deep recurrent neural networks using ant colony optimization. In: European Conference on Evolutionary
Computation in Combinatorial Optimization, pp. 86–98. Springer (2015)
• 4. Domhan, T., Springenberg, J.T., Hutter, F.: Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves.
In: IJCAI, vol. 15, pp. 3460–8 (2015)
• 5. Dorigo, M.: Optimization, learning and natural algorithms. PhD Thesis, Politecnico di Milano (1992). URL https://ci.nii.ac.jp/naid/10000136323/en/
• 6. Dorigo, M., Gambardella, L.M.: Ant colony system: a cooperative learning approach to the traveling salesman problem. IEEE Transactions on evolutionary
computation 1(1), 53–66 (1997)
• 7. ElSaid, A., Jamiy, F.E., Higgins, J., Wild, B., Desell, T.: Using ant colony optimization to optimize long short-term memory recurrent neural networks. In:
Proceedings of the Genetic and Evolutionary Computation Conference, pp. 13–20. ACM (2018)

28. ‫منابع‬
28
• 8. Elsken, T., Metzen, J.H., Hutter, F.: Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:1711.04528 (2017)
• 9. Google: https://colab.research.google.com
• 10. Jin, H., Song, Q., Hu, X.: Efficient neural architecture search with network morphism. arXiv preprint arXiv:1806.10282 (2018)
• 11. Krizhevsky, A., Nair, V., Hinton, G.: The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html p. 4 (2014)
• 12. LeCun, Y.: The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ (1998)
• 13. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436 (2015)
• 14. Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.J., Fei-Fei, L., Yuille, A., Huang, J., Murphy, K.: Progressive neural architecture search. In:
Proceedings of the European Conference on Computer Vision (ECCV), pp. 19–34 (2018)
• 15. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al.: Evolving deep neural
networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312. Elsevier (2019)
• 16. Miller, G.F., Todd, P.M., Hegde, S.U.: Designing neural networks using genetic algorithms. In: ICGA, vol. 89, pp. 379–384 (1989)
• 17. Negrinho, R., Gordon, G.: Deeparchitect: Automatically designing and training deep architectures. arXiv preprint arXiv:1704.08792 (2017)
• 18. Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018)
• 19. Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q., Kurakin, A.: Large-scale evolution of image classifiers. arXiv preprint
arXiv:1703.01041 (2017)

29. ‫منابع‬
29
• 20. Salama, K., Abdelbar, A.M.: A novel ant colony algorithm for building neural network topologies. In: International Conference on Swarm Intelligence, pp.
1–12. Springer (2014)
• 21. Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the
Genetic and Evolutionary Computation Conference, pp. 497–504. ACM (2017)
• 22. Wistuba, M., Rawat, A., Pedapati, T.: A survey on neural architecture search (2019)
• 23. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747
(2017)
• 24. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
• 25. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on
computer vision and pattern recognition, pp. 8697–8710 (2018)

30. Thank You
Amir shokri
Amirsh.nll@gmail.com
Dr. Kourosh Kiani