This document presents a distributed approach for solving partially flexible job-shop scheduling problems using cyber-physical products (CPPs) with Q-learning. CPPs use reinforcement learning to schedule jobs on heterogeneous machines with partial flexibility. The approach models a cyber-physical production system with intelligent CPPs that learn to schedule jobs in a distributed manner. Simulation results show the CPPs' scheduling performance improves over time as they learn through the Q-learning process.

1. A distributed approach solving partially flexible job-shop
scheduling problem with a Q-learning effect
Wassim BOUAZZA¹², Yves SALLEZ², Bouziane BELDJILALI¹
¹ LIO, Computer Sciences Department, University of Oran 1 Ahmed Ben Bella, ALGERIA
² LAMIH-CNRS, Department of Production Systems, University of Valenciennes & Hainaut-Cambrésis, FRANCE

2. Summary
2
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
Optimization Problem2
Proposed approach3
Experimentation4
Conclusion & Perspectives5
Context & Motivation1

3. Context & Motivation 3
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
Partial flexibility of a cell makes the scheduling more difficult, complicates the
search space, and increases the computation time (Kacem et al., 2002)
Deal with Partially Flexible Job-shop Scheduling Problem
Consider realistic constraints: Interoperability, times variations …etc
Heterarchical approach based on intelligent Cyber-Physical Product (CPP)
Q-Learning effect to reduce weakness of distributed approaches
Objectives
More complexity
CPPS
Cyber-Physical Production System
Industry 4.0

4. Optimization Problem: Scheduling problem & heterogeneous machine 4
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
A service can be processed on several alternative resourcesFJSP vs JSP
Total-FSP
Partial-FSP
• Family-dependent or Family-independent
• Sequence-dependent or Sequence-independent
Processing & Setup time
The FJSP solving consists on select a sequence of services and an assignment of
start/end times and resources for each service (Kacem et al., 2002)
Job families as pre-grouped jobs with same process requirements (Chen et al., 2013)

5. Optimization Problem: Scheduling in a Dynamic Environment 5
Well adapted for small-sized problems
Good Long-term optimization
Inefficient and impractical for solving large-sized problems owing
to the increased computation time requirement (Joo & Kim, 2015)
Don’t deal well with perturbation
Produce a reactive response to face dynamic perturbation
The decisions are then local and mainly do not go along with
global performance of the system
This phenomenon, due to lack of visibility of the autonomous
entities, is also called myopia (Zambrano Rey et al., 2014)
Use the past experience to reduce myopic phenomena by adding a Q-Learning technic
Distributed approachesCentralized approaches VS
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

6. j
Manufacturing Cell
Proposed Approach: CPPS developed 6
PhysicalLevelSoftwareLevel
Cyber-
physical
Product
Service
Provider
D
D
D
D
D
D
Decisional part Physical Product Resources
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
Traditionally, in the JSP, the assignment of
operations to the SP is not a priori fixed. That
is why many papers used a two-phase
method to face the FJSP. (Trentesaux et al.,
2013)
Learning cyber-physical products in manufacturing systems provide good opportunities for the future. The cyber-
physical product coupled with machine learning method offers new chances to increase the product’s performance in
term of flexibility and reactivity. (Bouazza et al., 2015)

7. Proposed Approach: Identifying the scheduling context 7
Families
SP1 SP2 SP3
P S P S P S
1 - - 5 - - -
2 6 2 4 2 5 2
3 5 2 5 2 5 2
1Processing Time 2Setup Time
Total
Partial
Single machine
Flexibility (FCi)
Without
Homogenous
Heterogeneous
Homogenous
Resource-dependent
Family-dependent
Processing Time (PTCi) Setup Time (STCi)
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

8. Proposed Approach: Reinforcement Learning (QAlgo) 8
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
Cyber-Physical Product
Process Controller1
Context Analysis
& Identification
Assignment Module
2
Sequencing Module
Scheduler3
A
B
Manufacturing
Information System
Stochastic
parameters
Knowledge
Database
Stochastic
parameters
Q1 Table
Q2 Table
Reinforcing
4 Waiting for service completion
Post-Decisional Evaluation5
a1∈ {SQ, LQE, SPT, SST}
a2∈ {FIFO, SJF, HPF, LIFO}
Weighted Average Waiting Time=∑(wjWtj)/J
Internal model of CPP
Qt+1(St,A)=(α-1)Qt(St,A)+α(Rt+1+γQt(St,A))
Learning rates Learning speed

9. Experimentation: Simulation tool developed 9
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
GUI of the MAS simulator developped
Manufacturing process
CPP parameters
Decisional statistics

10. Experimentation: Experimental data 10
Families
SP1 SP2 SP3 SP4 SP5 SP6
P1 S2 P S P S P S P S P S
1 2 5 - - - - - - - - - -
2 - - 3 - - - - - - - - -
3 - - - - 3 6 - - - - - -
4 - - - - 4 6 - - - - - -
5 3 - 3 - 3 - 3 - 3 - 3 -
6 4 2 4 5 4 4 4 6 4 7 4 4
7 - - - - - - - - - - 4 5
8 - - - - - - - - - - 5 -
9 - - - 5 - - - - 5 5 8 5
1Processing Time 2Setup Time
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
1. All SPs are assumed to be available at time 0.
2. All CPPs arrive dynamically.
3. Each CPP is assumed to have a priority (or criticality) that is a priori fixed.
4. Each SP has an input queuing zone, which is assumed to be infinite.
5. Each SP can process only one service at a time.
6. Once a service begins on an SP, it cannot be interrupted.
7. The availabilities and characteristics of SPs are supposed to remain
unchanged.
Assumptions
• Number of CPPs: J=500, j ∈ [1... 500]
• Number of families: F=9, f ∈ [1...9]
• Priority range: wj ∈ [1...20]
• CPP arrival times: Aij ∈ [1… 20999]
• CPP arrival rate: 1 CPP per 2 time units
Input Data

11. Experimentation: Results 11
Performance indicators
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
Machine Selection Rules distribution
Dispatching Rules distribution
16 combinations of MSR x DR
10 Executions of QAlgo

12. Conclusion & Perspectives 12
• The scheduling of partially flexible job shop is a complex issue, especially in a dynamic environment.
• A model of heterarchical Cyber-Physical Production System was presented.
• Q-learning associated with an original contextualization make the problem "dynamically" redefined by CPP.
• The use of learning techniques allows to enhance the global performance of the cyber-physical system.
• Thus, the CPP can cope with these complicated scheduling problems in an efficient decentralized way.
A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
• Those initial results encourage us to continue exploring this research way.
• Work is already underway to extend the approach with multiple production stages.
• It seems interesting to confront this method with even more realistic constraints: simultaneous production
tasks and failures.
• Comparative studies with metaheuristics as Genetic Algorithms or Particle Swarm Optimization.

13. Thanks for your attention
13A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -

14. 14A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect - IFAC’17 -
Bouazza, W., Sallez, Y., Aissani, N. and Beldjilali, B. (2015) ‘A model for manufacturing scheduling optimization through learning intelligent
products’, in Studies in Computational Intelligence. Springer International Publishing, pp. 233–241. doi: 10.1007/978-3-319-15159-5_22.
Chen, G., Li, M. and Kotz, D. (2008) ‘Data-centric middleware for context-aware pervasive computing’, Pervasive and Mobile Computing, 4(2), pp.
216–253. doi: 10.1016/j.pmcj.2007.10.001.
Joo, C. M. and Kim, B. S. (2015) ‘Hybrid genetic algorithms with dispatching rules for unrelated parallel machine scheduling with setup time and
production availability’, Computers & Industrial Engineering, 85, pp. 102–109. doi: 10.1016/j.cie.2015.02.029.
Kacem, I., Hammadi, S. and Borne, P. (2002) ‘Approach by localization and multiobjective evolutionary optimization for flexible job-shop
scheduling problems’, IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 32(1), pp. 1–13. doi:
10.1109/TSMCC.2002.1009117.
Trentesaux, D., Pach, C., Bekrar, A., Sallez, Y., Berger, T., Bonte, T., Leitão, P. and Barbosa, J. (2013) ‘Benchmarking flexible job-shop scheduling
and control systems’, Control Engineering Practice, 21(9), pp. 1204–1225. doi: 10.1016/j.conengprac.2013.05.004.
Zambrano Rey, G., Bonte, T., Prabhu, V. and Trentesaux, D. (2014) ‘Reducing myopic behavior in FMS control: A semi-heterarchical simulation-
optimization approach’, Simulation Modelling Practice and Theory, 46(0), pp. 53–75. doi: 10.1016/j.simpat.2014.01.005.