In robotic manipulation, benchmarks are crucial for analyzing experimental results against a common basis, and therefore making it easier to quantitatively interpret the performance of the methods. Our aim in this workshop is to generate specific benchmarking protocols for various sub-communities within robot manipulation. Rather than having a general discussion on the principles of benchmarking in robotics, this workshop will strictly concentrate on a concrete outcome: written benchmarks that are ready to be applied by the researchers.

We are forming workgroups targeting manipulation problems in several robotics sub-communities including grasp planning, force/tactile sensing for manipulation, machine learning for manipulation, non-prehensile manipulation, motion planning for manipulation, in-hand manipulation, gripper/end effector assessment. Each workgroup consists of researchers from different research laboratories. We believe that such a collaborative approach would help us cover the large array of interests in the community and reduce the risk of the benchmarks overfitting to a particular hardware or approach.

In the workshop, the benchmarks developed by the workgroups will be presented and discussed. The feedback of the community will be essential to finalize the benchmarks before they are officially published for use. We also submitted a proposal to an international robotics journal for publishing the finalized benchmarks in a special issue.

Please email to berk.calli@yale.edu for any questions or comments.