Formation of the IEEE-RAS Technical Committee on Performance Evaluation and Benchmarking for Robotic and Automation Systems (TC-PEBRAS)
(TC-PEBRAS) will serve as a forum to address performance evaluation and benchmarking for robotic and automation systems, in general. Currently, there is no accepted standard for quantitatively measuring the performance of intelligent systems – particularly robotic or automation systems - against user-defined requirements; and furthermore, there is no consensus on what objective evaluation procedures need to be followed to deduce the performance of these systems. The lack of reproducible and repeatable test methods has precluded researchers working towards a common goal from exchanging and communicating results, inter-comparing robot performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer. This lack of cohesion in the community hinders progress in many domains, such as manufacturing, service, healthcare, and security. Quantifying performance via benchmarking and standardization will improve the utility of robotic and automation systems in already established application areas. Having agreed-upon measures of performance and merit is a critical pre-requisite to wider acceptance and proliferation of emerging technologies. It is our hope that the efforts of this technical committee will bring together what is currently an amorphous community to provide a baseline for comparison and mechanisms for targeting specific aspects of a system thus allowing researchers and practitioners to assess the performance of various systems in different scenarios and environmental conditions. The end-user communities can use this forum to monitor progress in emerging technologies and to provide input regarding their needs and requirements. Benchmark and challenge problems in specific technology areas can be submitted to this TC to stimulate progress in concrete and directed ways.
TC-PEBRAS will concentrate on performance aspects of intelligent systems including software and other influencing factors. We expect to learn from the experiences and insights of the members of the existing TCs gained both through their theoretical development and practical implementation of intelligent systems in a variety of diverse application domains.
We welcome you to join this effort and contribute in any way possible!
For a list of past and upcoming activities, please visit the TC website at:
*NOTE* The co-chairs of the TC will be available during IROS'09 to meet with interested folks to answer any questions. Please email us to set up a mutually convenient time at:
(back to overview)