Resource-constrained production scheduling problem with multi-robot task in smart factory using deep reinforcement learning

Toan Thanh Phan; Tuan Anh Nguyen; Thoai Minh Vu

doi:10.15625/2525-2518/23452

Author affiliations

Authors

Toan Thanh Phan Thang Long University, Nghiem Xuan Yem Street, Dinh Cong Ward, Ha Noi, Viet Nam https://orcid.org/0000-0001-7230-2890
Tuan Anh Nguyen Thang Long University, Nghiem Xuan Yem Street, Dinh Cong Ward, Ha Noi, Viet Nam https://orcid.org/0000-0002-9939-5318
Thoai Minh Vu Thang Long University, Nghiem Xuan Yem Street, Dinh Cong Ward, Ha Noi, Viet Nam

DOI:

https://doi.org/10.15625/2525-2518/23452

Keywords:

reinforcement learning, smart factory, resource-constrained projects scheduling problem, automated guided vehicles, machine learning, neural networks, artificial intelligence

Abstract

Smart factories integrate diverse components such as humans, robots, artificial intelligence, cloud computing, and IoT systems, promising to enhance production efficiency, customization, and waste reduction to meet the demanding requirements of modern industry. A critical component is the Manufacturing Execution System (MES), which bridges the planning and production execution phases and performs functions including data collection, production coordination, equipment maintenance, detailed scheduling, resource allocation, and quality control. However, most current MES systems have not yet integrated resource scheduling capabilities, creating an urgent need for efficient scheduling algorithms that increase productivity, reduce workflow execution time, and optimize resource utilization. The core challenges involve multiple multi-stage tasks with complex constraints on completion time, system resources, and spatial positioning, including managing job allocation for multiple heterogeneous robots, coordinating resources to avoid delays from buffer limitations or bottlenecks, and optimizing execution time to prevent production-line stoppages. This research develops a machine learning model based on reinforcement learning to optimize resource-constrained production scheduling in smart factories, designated DRL-RCPSSP, focusing on improving performance, resource allocation, and adaptability under dynamic conditions.

Downloads

Download data is not yet available.

References

1. Vespoli S., Mattera G., Marchesano M. G., Nele L., Guizzi G. – Adaptive manufacturing control with deep reinforcement learning for dynamic WIP management in industry 4.0. Comput. Ind. Eng., 202 (2025) 110966. https://doi.org/10.1016/j.cie.2025.110966.

2. Blazewicz J., Lenstra J. K., Rinnooy Kan A. H. G. – Scheduling subject to resource constraints: Classification and complexity. Discrete Appl. Math., 5 (1983) 11–24. https://doi.org/10.1016/0166-218x(83)90012-4.

3. Graves S. C. – A review of production scheduling. Oper. Res., 29 (1981) 646–675. https://doi.org/10.1287/opre.29.4.646.

4. Khadivi M., Charter T., Yaghoubi M., Jalayer M., Ahang M., Shojaeinasab A., Najjaran H. – Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions. Comput. Ind. Eng., 200 (2025) 110856. https://doi.org/10.1016/j.cie.2025.110856.

5. Yuan E., Cheng S., Wang L., Song S., Wu F. – Solving job shop scheduling problems via deep reinforcement learning. Appl. Soft Comput., 143 (2023) 110436. https://doi.org/10.1016/j.asoc.2023.110436.

6. Zhang Y., Zhu H., Tang D., Zhou T., Gui Y. – Dynamic job shop scheduling based on deep reinforcement learning for multi-agent manufacturing systems. Robot. Comput.-Integr. Manuf., 78 (2022) 102412. https://doi.org/10.1016/j.rcim.2022.102412.

7. Burggräf P., Wagner J., Saßmannshausen T., Ohrndorf D., Subramani K. – Multi-agent-based deep reinforcement learning for dynamic flexible job shop scheduling. Procedia CIRP, 112 (2022) 57–62. https://doi.org/10.1016/j.procir.2022.09.024.

8. Takenaka T., Ashima A., Nishino N. – Strategies for evolving IoT-based product–service systems from emergent synthesis perspective. Procedia CIRP, 112 (2022) 1–5. https://doi.org/10.1016/j.procir.2022.09.014.

9. Zhang C., Song W., Cao Z., Zhang J., Tan P. S., Xu C. – Learning to dispatch for job shop scheduling via deep reinforcement learning. In: 34th Conference on Neural Information Processing Systems (NeurIPS 2020). Vancouver, Canada, (2020).

10. Myszkowski P. B., Laszczyk M., Nikulin I., Skowroński M. – iMOPSE: a library for bicriteria optimization in multi-skill resource-constrained project scheduling problem. Soft Comput., 23 (2018) 3397–3410. https://doi.org/10.1007/s00500-017-2997-5.

11. Hozdić E. – Smart factory for Industry 4.0: A review. Int. J. Mod. Manuf. Technol., 7 (2015) 28–35.

12. Modrak V., Sudhakarapandian R., Balamurugan A., Soltysova Z. – A review on reinforcement learning in production scheduling: An inferential perspective. Algorithms, 17 (2024) 343. https://doi.org/10.3390/a17080343.

13. Dong P., Xie J., Tang W., Xiong N., Zhong H., Vasilakos A. V. – Performance evaluation of multipath TCP scheduling algorithms. IEEE Access, 7 (2019) 29818–29825. https://doi.org/10.1109/access.2019.2898110.

14. Klein R. – Resource-constrained scheduling problems. In: Scheduling of Resource-Constrained Projects. Springer US, (2000) 73–109. https://doi.org/10.1007/978-1-4615-4629-0_3.

15. K.Suri P., Mittal S. – Design of stochastic simulator for analyzing the impact of scalability on CPU scheduling algorithms. Int. J. Comput. Appl., 49 (2012) 4–9. https://doi.org/10.5120/7717-1065.

16. Yohanes K. N. – Dual sourcing strategy in mass customized manufacturing. In: 2008 IEEE International Conference on Industrial Engineering and Engineering Management. IEEE, (2008) 1088–1092. https://doi.org/10.1109/ieem.2008.4738038.

17. Ghassemi Tari F., Olfat L. – Heuristic rules for tardiness problem in flow shop with intermediate due dates. Int. J. Adv. Manuf. Technol., 71 (2013) 381–393. https://doi.org/10.1007/s00170-013-5478-8.

18. Modrak V., Pandian R. S. – Flow shop scheduling algorithm to minimize completion time for n-jobs m-machines problem. Tehn. Vjesn., 17 (2010) 273–278.

19. Thenarasu M., Rameshkumar K., Rousseau J., Anbuudayasankar S. P. – Development and analysis of priority decision rules using MCDM approach for a flexible job shop scheduling: A simulation study. Simul. Model. Pract. Theory, 114 (2022) 102416. https://doi.org/10.1016/j.simpat.2021.102416.

20. Sadegheih A. – Scheduling problem using genetic algorithm, simulated annealing and the effects of parameter values on GA performance. Appl. Math. Model., 30 (2006) 147–154. https://doi.org/10.1016/j.apm.2005.03.017.

21. Pongchairerks P. – Particle swarm optimization algorithm applied to scheduling problems. ScienceAsia, 35 (2009) 89–94. https://doi.org/10.2306/scienceasia1513-1874.2009.35.089.

22. Sunday A. P., Emeka U. C., Chukwumuanya E. O., Chikwendu O. C. – Machine learning applications for production scheduling optimization. J. Explor. Dyn. Probl., 2 (2025) 63–79. https://doi.org/10.31004/edp.v2i4.137.

23. Cao F., Servranckx T., Vanhoucke M., He Z. – A comparison of different clustering algorithms for the project time buffering problem. Comput. Ind. Eng., 199 (2025) 110752. https://doi.org/10.1016/j.cie.2024.110752.

24. Jędrzejowicz P., Ratajczak-Ropel E. – Reinforcement learning strategy for A-Team solving the resource-constrained project scheduling problem. In: Computational Collective Intelligence. Technologies and Applications. Springer Berlin Heidelberg, (2013) 457–466. https://doi.org/10.1007/978-3-642-40495-5_46.

25. Mao H., Schwarzkopf M., Venkatakrishnan S. B., Meng Z., Alizadeh M. – Learning scheduling algorithms for data processing clusters. In: Proceedings of the ACM Special Interest Group on Data Communication. ACM, (2019) 270–288. https://doi.org/10.1145/3341302.3342080.

26. Choi J., Realff M. J., Lee J. H. – A Q-learning-based method applied to stochastic resource constrained project scheduling with new project arrivals. Int. J. Robust Nonlinear Control, 17 (2007) 1214–1231. https://doi.org/10.1002/rnc.1164.

27. Golab A., Gooya E. S., Falou A. A., Cabon M. – A convolutional neural network for the resource-constrained project scheduling problem (RCPSP): A new approach. Decis. Sci. Lett., 12 (2023) 225–238. https://doi.org/10.5267/j.dsl.2023.2.002.

28. Teichteil-Königsbuch F., Povéda G., González de Garibay Barba G., Luchterhand T., Thiébaux S. – Fast and robust resource-constrained scheduling with graph neural networks. In: Proceedings of the International Conference on Automated Planning and Scheduling, 33. Association for the Advancement of Artificial Intelligence (AAAI), (2023) 623–633. https://doi.org/10.1609/icaps.v33i1.27244.

29. Devanga A., Badilla E. D., Dehghanimohammadabadi M. – Applied reinforcement learning for decision making in industrial simulation environments. In: 2022 Winter Simulation Conference (WSC). IEEE, (2022) 2819–2829. https://doi.org/10.1109/wsc57314.2022.10015282.

30. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. – Proximal policy optimization algorithms 2017.

31. Zhao X., Song W., Li Q., Shi H., Kang Z., Zhang C. – A deep reinforcement learning approach for resource-constrained project scheduling. In: 2022 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE / IEEE, (2022) 1226–1234. https://doi.org/10.1109/ssci51031.2022.10022122.

32. Cai H., Bian Y., Liu L. – Deep reinforcement learning for solving resource constrained project scheduling problems with resource disruptions. Robot. Comput.-Integr. Manuf., 85 (2024) 102628. https://doi.org/10.1016/j.rcim.2023.102628.

33. Nakamura K., Derbel B., Won K.-J., Hong B.-W. – Learning-rate annealing methods for deep neural networks. Electronics, 10 (2021) 2029. https://doi.org/10.3390/electronics10162029.

34. Patterson A., Liao V., White M. – Robust losses for learning value functions. IEEE Trans. Pattern Anal. Mach. Intell. (2022) 1–12. https://doi.org/10.1109/tpami.2022.3213503.

35. Chen Q., Zhang Q., Liu Y. – Balancing exploration and exploitation in episodic reinforcement learning. Expert Syst. Appl., 231 (2023) 120801. https://doi.org/10.1016/j.eswa.2023.120801.