Benefits for the users

Collaboration between humans and industrial robots can relieve humans from tedious tasks that are more suited for robot execution. Vision is a convenient modality and interface for interaction/coordination as it does not require any contact and can be activated from a distance. This use-case demonstrates capabilities for vision-based collaboration between human and robot, by visual perception tools such as human skeleton detection, human action recognition and the detection and pose estimation of objects and targets in the scene.

Innovation

Perception and situational awareness of robot systems can be enhanced, such that fluent and responsive collaboration between human and robot is possible. Perception models, based on deep learning, are ideal for this, as they can be accurate, reliable and fast to execute.

Risks and limitations

Software Malfunction: As the whole system is operates by a computer, there are chances that software and connections through devices get malfunctioned. These can endanger the operator’s safety while the system is not stopped. As a result, it requires a back-up safety system running all the time and malfunctioning of this system should result in protective stop of robot system. Environmental disturbances: lighting conditions, dust electrical interferences can affect the detection accuracy of the visual tools, which can cause false or mis-detections.

Technology readiness level

5

Sectors of application

Manufacturing .

Potential sectors of application

Any sector where collaboration between human and robot would occur, e.g., inspection and maintenance, agrofood, healthcare

Patents / Licenses / Copyrights

Hardware / Software

Hardware:

Franka Emika collaborative robot Standard RGB camera (e.g. Intel Realsense D435) Workstation with or without GPU running Linux Ubuntu 20.04 LTS

Software:

Linux Ubuntu 20.04 LTS ROS1 Noetic or ROS2 Foxy OpenDR toolkit will install all required dependencies: https://github.com/opendr-eu/opendr/

Video