Human Activity Recognition for Security Check & Hand Washing

Problem

Industries rely on safety and cleanliness checks, but there is no guarantee every employee performs them to the standard, and placing an expert to watch everyone does not scale. A computer can check automatically, but it first has to recognize the motion. Frame-level features from a 2D CNN miss the temporal pattern of an activity.

Approach

Instead of feeding whole frames to the model, we turned each frame into OpenPose keypoints: 25 body points for security check, and 21 points per hand for hand washing, each with x, y, and a confidence value. The keypoints are flattened, stacked into 20-frame sequences with a frame lag of 2, and classified by an LSTM (256 units) followed by dense layers. We recorded our own 1080p, 30 FPS dataset: 150 security-check videos and 240 hand-washing videos from 5 people across 7 activities.

How It Works

1. 1
   RecordActivity videos at 1920×1080 and 30 FPS, one person per clip.
2. 2
   Extract poseOpenPose gives body keypoints (security check) or both hands’ keypoints (hand wash) per frame.
3. 3
   Build sequencesFlatten the keypoints into 20-frame sequences. A frame lag of 2 adds the previous frames to each step, giving a 3D input (samples × timesteps × features).
4. 4
   ClassifyLSTM (256) → dense 128 → 64 → 32 (ReLU) → softmax over the activities.
5. 5
   EvaluateRandom 80:20 train/test split made after building the frame-lag sequences, RMSProp with cross-entropy for 100 epochs, then tests on video and a live camera.

What I Built

• Two self-recorded datasets: security check (rub head, rub left hand, rub right hand, rub feet) and hand washing (rub hands, rub backhand, splash hands).
• An OpenPose keypoint pipeline that turns raw video into flattened, frame-lagged pose sequences.
• An LSTM activity classifier per use case, trained and tested in Keras/TensorFlow.
• A demo that recognizes activities from recorded video and a live camera.

Impact

The model reached 98% test accuracy on security check and 63% on hand washing, showing that pose keypoints are enough to recognize full-body routines. Hand washing was harder because not every keypoint carries a major movement. Our recommendations were feature selection, better camera angles, more varied data, and checking that the water is actually flowing.