Towards Applied Experiment and System Reproduction in Robotics

The corresponding paper has been published and presented at IROS 2016

Even though research on autonomous robots and human-robot interaction accomplished notable progress in recent years, and sophisticated reusable soft- and hardware components are available, many of the reported findings are only hardly reproducible by fellow scientists. This is usually due to the fact that crucial information, such as the specification of used software versions and their configuration, required data sets, and experiment protocols are omitted in most publications. In order to address these issues, we recently introduced an integrated software tool chain to facilitate experiment reproduction in robotics. In this contribution we instantiate the complete tool chain in a — so far — unique user study in order to assess its applicability and usability. For this, we chose three different robotic systems from independent institutions and modeled them in our tool chain, including three purposefully limited experiments. Subsequently, we asked twelve researchers to reproduce one of the formerly unknown systems and the associated experiment. We could show that all non-experts were able to replicate a formerly unknown robotics experiment using our approach.

This page shall provide all necessary information to also make the study itself replicable and traceable.

Replicated Systems

Replicated Experiments

Expected Outcome



 

Experiment Instructions

Below you can find the original instruction PDFs that were given to the subjects. We removed the link to the questionnaire and, in order to provide the original CITK repository state at the time of the study, we added a git commit hash to the "git clone" command and freezed the jenkins.tar.gz version.

Raw Study Results

Below you can find the results and according questions of this study's questionnaire. Please feel free to confirm our results. We removed the timestamp and the field of study in order to maintain anonymization of the subjects.

Exemplary Stud Trial Video

Note

Right now, you will need credentials in order to checkout parts of the Flobi System, please let us know if you are about to replicate the Flobi experiment. However this will become obsolete as we are about to release the software under an open source license any time soon.