14 May 2019
First internal presentation of BenchBot concept to ACRV review panel, focusing only on real robots
15 November 2019
Internal alpha release of BenchBot as a bridge between simulation and real robots (v0.1.0)
20 March 2020
Public beta release for testing (v0.5.0)
9 June 2020
First mainstream release, ready for public use (v1.0.0)
3 August 2020
Preprint released on arXiv describing the BenchBot software suite
17 February 2021
Support for more research problems, simulators, and robot platforms, through a new modular architecture powered by add-ons (v2.0.0)
17 February 2021
Robotic Vision Scene Understanding (RVSU) challenge released, powered by BenchBot, as part of the Embodied AI workshop at CVPR 2021
14 February 2022
Robotic Vision Scene Understanding (RVSU) challenge released, powered by BenchBot, as part of the Embodied AI workshop at CVPR 2022
18 February 2022
First version released with support for NVIDIA's Omniverse-powered Isaac Sim (v2.2.0)
2 March 2022
BenchBot environments for active robotics (BEAR) published in IJRR
26 May 2022
Technical blog featuring BenchBot released on NVIDIA's developer website
The BenchBot software stack is a collection of software packages that allow end users to control robots in real or simulated environments with a simple python API. It leverages the simple "observe, act, repeat" approach to robot problems prevalent in reinforcement learning communities (OpenAI Gym users will find the BenchBot API interface very similar).
BenchBot was created as a tool to assist in the research challenges faced by the semantic scene understanding community; challenges including understanding a scene in simulation, transferring algorithms to real world systems, and meaningfully evaluating algorithm performance. We've since realised, these challenges don't just exist for semantic scene understanding, they're prevalent in a wide range of robotic problems.
This led us to create version 2 of BenchBot with a focus on allowing users to define their own functionality for BenchBot through add-ons. Want to integrate your own environments? Plug-in new robot platforms? Define new tasks? Share examples with others? Add evaluation measures? This all now possible with add-ons, and you don't have to do anything more than add some YAML and Python files defining your new content.
Shortly after the release of version 2, NVIDIA began publicising their Omniverse platform as a major step forward in the 3D simulation and digital design landscape. This included a major upgrade to their Isaac Sim robotics simulator. The upgrade brought significant improvements to simulation realism, with the below images highlighting better reflection and rendering capabilities respectively.
In consultation with NVIDIA, we upgraded BenchBot to move from the old Unreal Engine-powered simulations to Omniverse. The upgrades bought technical challenges, improvements to BenchBot's robot platform integrations, and ultimately a deeper collaboration with NVIDIA's Embedded AI team. This collaboration led to me writing a technical blog post for NVIDIA, which was featured on the front page of their developers site.
The main BenchBot repository contains the software stack needed to develop solutions for BenchBot tasks on your local machine. It installs and configures a significant amount of software for you, wraps ~50GB of software in stable Docker images, and provides simple interaction with the stack through 4 basic scripts:
The stack also handles installing our set of semantic scene understanding (SSU) add-ons by default. These add-ons can be swapped out, expanded on, or replaced with your own. Documentation for the add-ons system can be found in the benchbot_addons repository.
We've used BenchBot in a number of places already in our research, including our semantic scene understanding challenges. See below for an overview of how the challenges work.
The "bench" in "BenchBot" refers to benchmarking, with our goal to provide a system that greatly simplifies the benchmarking of novel algorithms in both realistic 3D simulation and on real robot platforms. If there is something else you would like to use BenchBot for (like integrating different simulators), please let us know. We're very interested in BenchBot being the glue between your novel robotics research and whatever your robot platform may be.
© Ben Talbot. All rights reserved.