Implementation of Finger Gesture-Based Medicine Delivery Robot Control System with MediaPipe

Ahmad Zarkasi, Gede Pradnyananda, Sarmayanta Sembiring, Aditya P. P. Prasetyo, Ricy Ricy Firnando, Abdurahman Abdurahman

Abstract

This research introduces a prototype drug delivery robot that is controlled entirely through finger gestures, without the need for direct touch or additional physical control devices. The system utilizes a Pi Camera connected to a Raspberry Pi to capture the image of the user's hand, which is then processed by MediaPipe Hands to perform detection and extraction of 21 point landmarks in real-time. The position and relationship between landmarks are converted into a five-bit binary vector that represents which finger is raised. This binary data is then sent via serial communication to the STM32 Nucleo microcontroller, which is tasked with translating the binary pattern into motion commands (forward, backward, turn right/left, stop) to drive the DC motor. The results of testing the gesture recognition system, which was performed 30 times for each command, showed a high success rate. The gesture Stop achieved perfect success (30/30), followed by 'Forward' and 'Right' with 10 successes, and 'Backward' and 'Left' with 9 successes. This test shows that the system is able to respond accurately to gesture commands at a distance of 20 to 250 cm. The robot was also able to execute all motion commands responsively and accurately according to the recognized gestures. This prototype proves that MediaPipe can be an efficient and reliable method to implement gesture-based robot control on resource-constrained embedded platforms, as well as potentially applied in healthcare environments to minimize physical contact.

Full Text:

PDF

References

[1] J. Qi et al., “Computer vision-based hand gesture recognition for human-robot interaction: a review,” Complex & Intelligent Systems, vol. 10, pp. 1581–1606, 2024, doi: 10.1007/s40747-023-01173-6.

[2] A. Sharma et al., “Hand Gesture Recognition using Image Processing and Feature Extraction Techniques,” Procedia Computer Science, vol. 173, pp. 181–190, 2020, doi: 10.1016/j.procs.2020.06.022.

[3] A. O. Hashi et al., “A Systematic Review of Hand Gesture Recognition: An Update From 2018 to 2024,” IEEE Access, vol. 12, pp. 143599–143623, 2024 doi: https: 10.1109/ACCESS.2024.3421992.

[4] F. Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking,” Google Research, 2020,doi : 10.48550/arXiv.2006.10214.

[5] S. Kavirayani et al., “Robot for Delivery of Medicines to Patients Using Artificial Intelligence in Health Care,” in 2021 IEEE Conference Publications, doi: doi.org/10.1109/B-HTC50970.2020.9297948.

[6] V. Moysiadis et al., “An Integrated Real-Time Hand Gesture Recognition Framework for Human–Robot Interaction in Agriculture,” Applied Sciences, vol. 12, no. 8160, 2022,doi: 10.3390/app12168160.

[7] N. Mohamed, M. B. Mustafa, and N. Jomhari, "A Review of the Hand Gesture Recognition System: Current Progress and Future Directions," IEEE Access, vol. 9, pp. 157422–157437, 2021, doi: 10.1109/ACCESS.2021.3129650

[8] A. M. Chalik, B. A. Qowy, F. Hanafi, and A. Nuraminah, "Mouse Tracking Tangan dengan Klasifikasi Gestur Menggunakan OpenCV dan Mediapipe," J. Ilm. Tek. Inform. dan Komun., vol. 1, no. 2, pp. 10–18, 2021

[9] D. Khairianto and R. Firdaus, "Penerapan Hand Gesture Recognition sebagai Media Kontrol Presentasi Aplikasi PowerPoint," JATI (Jurnal Mahasiswa Teknik Informatika), vol. 8, no. 2, pp. 1852–1860, 2024,doi: 10.36040/jati.v8i2.9167.

[10] H. Susilawati and F. Nuraeni, "Raspberry Based Hand Gesture Recognition Using Haar Cascade and Local Binary Pattern Histogram," Jurnal Ilmiah Teknik Elektro Komputer dan Informatika (JITEKI), vol. 8, no. 4, pp. 621–633, 2022, doi: 10.26555/jiteki.v8i4.24643

[11] Y. Zhang and G. Notni, "3D geometric features based real-time American sign language recognition using PointNet and MLP with MediaPipe hand skeleton detection," Measurement: Sensors, vol. 35, p. 101697, 2024, doi: 10.1016/j.measen.2024.101697

[12] M. R. Summerfield, F. J. Seagull, N. Vaidya, and Y. Xiao, "Use of pharmacy delivery robots in intensive care units," Am J Health-Syst Pharm, vol. 68, no. 1, pp. 77-83, 2011,doi:10.2146/ajhp100012.

[13] S. Senthilkumar, B. Dhanalakshmi, P. Shanmuga Prabha, A. Verma, E. Muniyandy, and S. Sridhar, "An IoT Intelligent Approach for Safety and Efficiency of Robotic Medicine Delivery in Hospitals," J. Electrical Systems, vol. 20, no. 2s, pp. 820-827, 2024,doi: 10.52783/jes.1677.

[14] H. Cao, X. Huang, J. Zhuang, J. Xu, and Z. Shao, "CIoT-Robot: Cloud and IoT Assisted Indoor Robot for Medicine Delivery," in 2018 Joint International Advanced Engineering and Technology Research Conference (JIAET 2018), Atlantis Press, 2018, pp. 85-89,doi: 10.2991/jiaet-18.2018.14.

[15] A. Bagade, A. Kulkarni, P. Nangare, P. Shinde, and S. Gudadhe, "Robotic Assistant for Medicine and Food Delivery in Healthcare," in Cyber Security, Privacy and Networking, D. P. Agrawal, N. Nedjah, B. B. Gupta, and G. M. Perez, Eds. Singapore: Springer, 2022, pp. 49-59,doi: 10.1007/978-981-16-8664-1_5.

[16] F. Umam, "Pengembangan Sistem Kendali Pergerakan Autonomous Mobile Robot untuk Mendapatkan Jalur Bebas Hambatan Menggunakan Fuzzy Logic Controller," Jurnal Ilmiah Mikrotek, vol. 1, no. 1, pp. 35-42, 2013

[17] A. Joy et al., "MedRobo: Medicine Delivering and Patient Parameter Monitoring Robot," in 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, 2021, pp. 1808-1812,doi: 10.1109/ICACCS51430.2021.9441692.

[18] M. Manigandan and I. Manju Jackin, "Wireless Vision based Mobile Robot control using Hand Gesture Recognition through Perceptual Color Space," in 2010 International Conference on Advances in Computer Engineering, Bangalore, India, 2010, pp. 95-99,doi: 10.1109/ACE.2010.69.

[19] S. S. Kannan, A. Lee, and B.-C. Min, "External Human-Machine Interface on Delivery Robots: Expression of Navigation Intent of the Robot," arXiv preprint arXiv:2108.03045, 2021,doi: https://doi.org/10.48550/arXiv.2108.03045

[20] M. Oudah, A. Al-Naji, and J. Chahl, "Hand Gesture Recognition Based on Computer Vision: A Review of Techniques," J. Imaging, vol. 6, no. 8, p. 73, 2020,doi: 10.3390/jimaging6080073.

DOI: https://doi.org/10.20961/jeeict.8.1.108142

Refbacks

  • There are currently no refbacks.