Sensor-based Human-Robot Collaboration

Name of demonstration

Sensor-based Human-Robot Collaboration

Main objective

Demonstrating capabilities for vision-based collaboration between human and robot. Deep learning-based perception tools are utilized to provide input for the collaboration.

Short description

Demonstration of a vision-based system for human-robot collaboration in the assembly of diesel engine components. Visual perception of a person, their actions is utilized for coordinating the shared task. Visual perception is also used to detect objects and targets and perform grasping actions for pick and placement, as well as robot-human hand-overs. 

Owner of the demonstrator

Tampere University

Responsible person

Roel Pieters


C29.3 - Manufacture of parts and accessories for motor vehicles


Robotics, Vision System, Machine Learning, human-robot collaboration, cobot assisted manufacturing, AI.

Benefits for the users

Collaboration between humans and industrial robots can relieve humans from tedious tasks that are more suited for robot execution. Vision is a convenient modality and interface for interaction/coordination as it does not require any contact and can be activated from a distance. This use-case demonstrates capabilities for vision-based collaboration between human and robot, by visual perception tools such as human skeleton detection, human action recognition and the detection and pose estimation of objects and targets in the scene.


Perception and situational awareness of robot systems can be enhanced, such that fluent and responsive collaboration between human and robot is possible. Perception models, based on deep learning, are ideal for this, as they can be accurate, reliable and fast to execute.

Risks and limitations

Software Malfunction: As the whole system is operates by a computer, there are chances that software and connections through devices get malfunctioned. These can endanger the operator’s safety while the system is not stopped. As a result, it requires a back-up safety system running all the time and malfunctioning of this system should result in protective stop of robot system. Environmental disturbances: lighting conditions, dust electrical interferences can affect the detection accuracy of the visual tools, which can cause false or mis-detections.

Technology readiness level


Sectors of application

Manufacturing .

Potential sectors of application

Any sector where collaboration between human and robot would occur, e.g., inspection and maintenance, agrofood, healthcare

Patents / Licenses / Copyrights
Hardware / Software


Franka Emika collaborative robot Standard RGB camera (e.g. Intel Realsense D435) Workstation with or without GPU running Linux Ubuntu 20.04 LTS


Linux Ubuntu 20.04 LTS ROS1 Noetic or ROS2 Foxy OpenDR toolkit will install all required dependencies:

Back to LIST
Subscribe to newsletter

Please subscribe to our mailing list to be the first one informed about the new open calls, events and other hot news!

    Please check if the email you entered is correct
    All done!

    Your have successfully subscribed to Trinity newsletter.