Skeleton based Human action recognition

Main functionalities

Skeleton-based human action recognition tool for recognizing human actions, in the field of view, from a detected human skeleton. Up to 60 actions human actions can be recognized, including single human actions and human to human interactions. The full list can be seen here: https://rose1.ntu.edu.sg/dataset/actionRecognition/ Method Recognition of human actions is done with ST-GCN (https://github.com/yysijie/st-gcn), a real-time skeleton-based human action recognition framework. Depending on the dataset the method can detect a large number of different human actions, ranging from daily activities to complex actions with interactions. It utilizes the lightweight OpenPose model, to estimate the location of the human joints in every image, to generate a sequence of detected human skeleton graphs, connected both spatially and temporally. Data generation and model training The utilized training dataset is NTU-RGB+D (https://rose1.ntu.edu.sg/dataset/actionRecognition/), which contains 56,000 human action clips in 60 human action classes. For each image human skeleton joints are annotated in 3D, with respect to the camera coordinate system. The pretrained model from the original authors, with default training parameters, is used for inference.

Technical specifications

RGB camera (e.g. Intel Realsense D435) connected to a workstation, running Ubuntu 20.04. Framerate of the camera ranges between 30-60 Hz, depending on the image resolution. ROS (Noetic or Foxy) is used as interface for communication. The OpenDR toolkit provides the tool and the interfacing to ROS. Other cameras will work, as long as the images are published on the ROS network. A demo example can be found here (ROS): https://github.com/opendr-eu/opendr/tree/master/projects/opendr_ws/src/perception#skeleton-based-human-action-recognition-ros-node A demo example can be found here (non-ROS): https://github.com/opendr-eu/opendr/blob/master/projects/perception/skeleton_based_action_recognition/REAMDE.md More information is explained here: https://github.com/opendr-eu/opendr/blob/develop/docs/reference/skeleton-based-action-recognition.md

Inputs and outputs

Input to the tool is given by a standard ROS image message, for example run by a webcam node: rosrun usb_cam usb_cam_node with the image topic: /usb_cam/image_raw Output of the tool are the predicted class id and confidence, published under the topic name /opendr/skeleton_based_action_recognition, and the human-readable class name under /opendr/skeleton_based_action_recognition_description. Besides, the annotated image is published in /opendr/image_pose_annotated as well as the corresponding poses in /opendr/poses.

Formats and standards

• Linux Ubuntu 20.04 LTS
• ROS1 Noetic or ROS2 Foxy
• OpenDR v2.0 
• Pytorch 1.9
• Python 3.8

All dependencies will be installed with the OpenDR toolkit and/or installation instructions are provided

Owner (organization)

Tampere University