One of the main goals of this website is to serve as a portal for proposition and discussion of new community driven protocols and benchmarks. The following links can guide you through proposing new protocols and benchmarks:
Protocol Template Benchmark Template
Protocol Template Explanations Benchmark Template Explanations
If you would like your protocol/benchmark added to this website, please email to Berk Calli (bcalli@wpi.edu)
A standard set of objects and associated models are a great starting point for common replicable research and benchmarking in manipulation, but protocols and benchmarks that define the experimental procedure and quantification methods are equally important for common replicable research and benchmarking in manipulation. Given the wide range of technical interests, research approaches and applications being examined in the manipulation research community, along with how quickly the field moves, it is quite challenging to provide sufficient task descriptions that will span the range of interests and remain relevant in the long-term. Therefore, we seek to lay the groundwork for those to be driven by the research community and sub-communities.
The proposed protocols and benchmarks by the community will be displayed here. The discussions on these protocols and benchmarks are in the corresponding forum page.
Here are the proposed protocols and benchmarks so far:
| Name | Publication Date | Authors | Purpose | Files (ID / Version) |
|---|---|---|---|---|
| The Elliott and Connolly Benchmark: A Test for Evaluating the In-Hand Dexterity of Robot Hands | Oct. 2021 | Ryan Coulson, Chao Li, Carmel Majidi, and Nancy S. Pollard | evaluating in- hand dexterity, specifically for humanoid type robot hands | Protocol Benchmark Videos Paper Supplementary |
| Benchmarking Protocol for Grasp Planning Algorithms | Feb. 2020 | Yasemin Bekiroglu, Naresh Marturi, Máximo A. Roa, Komlan Jean Maxime Adjigble, Tommaso Pardi, Cindy Grimm, Ravi Balasubramanian, Kaiyu Hang, Rustam Stolkin | Assess the performance of a grasp planning algorithm in a physical robot setup, minimizing the influence of key components in the grasping pipeline | Protocol Benchmark Videos Paper |
| Bimanual Manipulation Benchmark | Feb. 2020 | Konstantinos Chatzilygeroudis, Bernardo Fichera, Ilaria Lauzana, Fanjun Bu, Kunpeng Yao, Farshad Khadivar, and Aude Billard | Assess the motion planning and/or learning performance of a multi-arm systems | Protocol Benchmark Videos Paper |
| Benchmarks for Aerial Manipulation | Feb. 2020 | Alejandro Suarez , Victor M. Vega, Manuel Fernandez, Guillermo Heredia, Anibal Ollero | Evaluate and compare the performance of aerial manipulators | Protocol and Benchmarks Videos Paper |
| Box and Blocks Benchmark | Feb. 2020 | Andrew S. Morgan, Kaiyu Hang, Walter G. Bircher, Fadi M. Alladkani, Abhinav Gandhi, Berk Calli, Aaron M. Dollar | Assess unilateral gross manual dexterity | Protocol Benchmark Videos Supplementary Paper |
| Benchmarking In-Hand Manipulation | Feb. 2020 | Silvia Cruciani, Balakumar Sundaralingam, Kaiyu Hang, Vikash Kumar, Tucker Hermans, Danica Kragic | Evaluate the planning and control aspects of robotic in-hand manipulation systems | Protocol Benchmark Videos Paper |
| GRASPA-benchmark | Feb. 2020 | Fabrizio Bottarel, Giulia Vezzani | Assess the performance of a grasp planning algorithm in a physical robot setup, accounting for the limitations of the test platform | Protocol Benchmark Paper |
| Bimanual Cloth Manipulation Benchmark | Feb. 2020 | Irene Garcia-Camacho*, Martina Lippi*, Michael C. Welle, Hang Yin, Rika Antonova, Anastasiia Varava,Julia Borras, Carme Torras, Alessandro Marino, Guillem Alenya, Danica Kragic | Assess the ability of a bimanual robotic system to manipulate cloth pieces | Protocol Benchmark Videos Paper |
| Grasp Resilience Benchmark | Feb. 2020 | F. Negrello, W. Friedl, G. Grioli, M. Garabini, O. Brock, A. Bicchi, M. Roa, M. G. Catalano | Characterizing the behavior of robotic hands under impulsive loads conditions. | Reference Document Paper |
| Simulated Robotic Manipulation Protocol | Feb. 2020 | J. Collins, J. McVicar, D. Wedlock, R. Brown, D. Howard and J. Leitner | Assessing manipulation performance in simulation | Protocol Benchmark Videos Paper |
| Rubik's Cube Manipulation Protocol | Feb. 2020 | Boling Yang, Patrick Lancaster, Siddhartha S. Srinivasa, Joshua R. Smith | Evaluating a robotic system's overall manipulation accuracy, speed, and robustness. | Protocol Benchmark Videos Paper |
| The Anthropomorphic Hand Assessment Protocol (AHAP) | Nov. 2019 | Immaculada Llop-Harillo, Antonio Pérez-González, Julia Starke, Tamim Asfour (Universitat Jaume I, Karlsruhe Institute of Technology (KIT)) | Quantifying the grasping ability of artificial hands and comparing hand designs | Protocol Benchmark Paper |
| Gripper Assessment Protocol and Benchmark | Sept. 2015 | Berk Calli, Aaron Dollar (Yale University) | Assessing the abilities of a gripper to firmly grasp objects of different sizes and shapes. | Protocol Benchmark Paper |
| Table Setting Protocol and Benchmark | Sept. 2015 | Berk Calli (Yale University) | To assess the pick and place ability of a robotic system. | Protocol Benchmark Paper |
| Block Pick and Place Protocol and Benchmark | Sept. 2015 | Aaron Walsman, Siddhartha Srinivasa (CMU) | Assess the dexterity of a robotic manipulator using a pick-and-place task. | Protocol Benchmark Paper |
| Peg Insertion Learning Assessment Protocol and Benchmark | Sept. 2015 | Arjun Singh, Pieter Abbeel (UC Berkeley) | To assess the learning performance of a robot by the task of inserting a single peg to a hole. | Protocol Benchmark Paper |
| Pitcher-Mug Protocol and Benchmark | Sept. 2015 | Aaron Walsman, Berk Calli (CMU, Yale University) | To assess the ability of a robot to execute a common daily task. | Protocol Benchmark Paper |
Saving...