Prediction of Fatigue Using Deep Learning Models

A headache, pain and drowsiness are the most common symptoms of fatigue caused by a long duration of work using a visual display terminal (VDT). A sign of the headache generally involves placing a hand on the head, eyes, nose, or face. The recognition of these gestures is a challenging problem due to the difficulty in similar skin color of hands and face. Drowsiness detection is one of the interesting research on computer vision regarding driver’s fatigue. Previous studies of drowsiness detection have mainly focused on tracking of the eyes and mouth conditions like eyes blinking, yawing, and so on. The detection of hand gesture used in this work as the sign of drowsiness such as the hand cover on mouth and with the closure of the eyes by this hand gesture sign. The applications of this research can expand from only the observation of drowsiness for drivers to observing the status of drowsiness at other environments, for example, call center agents, control room observers, and air traffic controllers. Facial expressions and hand gestures recognized as a part of human emotions, especially in a feeling of fatigue signs. In computer vision research, a positioning of hand over face is one of the challenging problems. Therefore, the challenges summarize as follows: The difficulty of the difference of skin color for hands and face with the same texture, the limitation of illumination change, and the performance of traditional feature extraction methods.
In this research, modeling the symptom characteristics of fatigue like headaches, upper pain, and drowsy using the hand pose e.g. The hand on the forehead or covering the eye or nose and develop a system for recognizing the fatigue sign and alert the computer users to take rest and do some optimum exercise to staying healthy and safe. The proposed system is easy to install on a general PC because it only uses an RGB camera. There is no need for distinct gadgets and markers. Multiple methods used to improve the classification accuracy and avoid the face detection failure in the case of the hand over face gestures by performing the skin and face detection and the deep learning method for classification. In the proposed method, a deep learning based on a convolutional neural network (CNN) for the classification of the hand poses is applied. In addition, a class activation map (CAM) to visualize the prediction of the classification network for localization of the hand over face poses implemented. The CAM used for three tasks: the first one is to visualize the important area guided the prediction decision, it found the conflict caused by the error of prediction and modified the data and trained the network again until the higher classification accuracy without overfitting. The second task is to avoid under fitting by designing the deep network based on the visual explanation of prediction on every convolutional layer. In the third task, CAM used for localization of the hand over face poses. The proposed method was designed to reduce the type-I error (false-positive result) for the classification. The main Contributions in this research includes a framework for fatigue detection using computer vision and deep learning, computer vision Model for recognizing the sign of fatigue using hand pose, a method for hand localization without annotations dataset and a method to solve the hand over face occlusion problem. From the experimental results, the hands over face gestures achieved classified under illumination change (50–500 lx), head orientation (∼45◦ left or right) and solved the occlusion problem of the hand over face. The drowsiness successfully recognized in the following cases:
(1) The eyes closed and opened the mouth.
(2) The eyes closed, and the hand covered on the mouth. The system achieved the classification ratio of drowsiness average 96.6 % and pain signs average 99.5%.
A headache prediction the hand poses as the signs of frontal and unilateral headaches without the classification overfitting and data biasing errors successfully classified with high accuracy recognition ratio of 98.5% for classification of the hand over face poses as the prediction of headaches.
Finally, as a social impact, this research can help the computer vision and physical therapy researchers to design new algorithms to recognize the visible fatigue signs using images captured by computer cameras, mobile phones, surveillance cameras in the workplace, and care for the elderly, children and disability humans.

内容要旨・審査要旨・論文本文の公開