ODIN Digital twin using sensor data fusion

Main functionalities

ODIN project has developed an operating system of the digital twin to connect virtual and real HRC cells. A HRC system has been build implementing digital twin functionality which includes several components for Java 3D models of human-robot work cells, sensor data processing and fusion, and communication between the physical and digital twins.

Technical specifications

For network transport, the Digital Twin system uses a Transmission Control Protocol for the communication between the robot and the server, while HTTP streams are used for data sharing from the server to remote users. Sensor data from the robot are passed to the server in packets when a human operator opens a subscription to robot monitoring for a specific work area. The visualization includes not only the pose information and operation patterns of the monitored robot but also the movements of the human operator after being calibrated, ultimately enabling communication and connectivity for HRC in both virtual and physical spaces

A digital twin system based on the software of Visual Components is designed to reflect the real-time motions of human operators and robots. For the human operators, the digital human model with predefined hierarchy is set up in the software. The data collection hardware communicates with the software in real time via socket to synchronize the movements of physical operators with the digital operators. For the robots, the ABB IRB 120 and ABB IRB 1600 robot models are configured with associated communications between these models and controllers of physical robots.

In terms of the mapping from the detected human body skeleton to the ongoing motions of digital operator in the Visual Components software, a combination of forward joint angle calculation and inverse kinematics is designed to improve the robustness of task-oriented entire movements. In terms of the forward joint angle calculation, the bones are first represented as spatial vectors based on the coordinates of detected 25 key points (joints). Then the angle of two vectors along a specific axis is calculated by projecting the vectors into the plane normal to that axis and using the arccos function to derive the value.

Based on above calculations, some of the necessary angles are transferred to the digital operator in the Visual Components. A threshold of rotation range is set for each joint individually, so that the movements of digital operator adhere to the physical constraints of human body. In terms of the inverse kinematics, it is implemented for the upper part of human body. Backward calculations are performed from the spine to the joints on the left and right arms. The mechanism of calculation is similar to the inverse kinematics for the robot arms. Moreover, during the normal movements only forward joint angle calculation is enabled to drive the digital operator.

Owner (organization)

KTH