The BenchBot software stack is a collection of software packages that allow end users to control robots in real or simulated environments with a simple python API. It leverages the simple "observe, act, repeat" approach to robot problems prevalent in reinforcement learning communities (OpenAI Gym users will find the BenchBot API interface very similar).

The BenchBot Story

BenchBot was created as a tool to assist in the research challenges faced by the semantic scene understanding community; challenges including understanding a scene in simulation, transferring algorithms to real world systems, and meaningfully evaluating algorithm performance. We've since realised, these challenges don't just exist for semantic scene understanding, they're prevalent in a wide range of robotic problems.

This led us to create version 2 of BenchBot with a focus on allowing users to define their own functionality for BenchBot through add-ons. Want to integrate your own environments? Plug-in new robot platforms? Define new tasks? Share examples with others? Add evaluation measures? This all now possible with add-ons, and you don't have to do anything more than add some YAML and Python files defining your new content.

Shortly after the release of version 2, NVIDIA began publicising their Omniverse platform as a major step forward in the 3D simulation and digital design landscape. This included a major upgrade to their Isaac Sim robotics simulator. The upgrade brought significant improvements to simulation realism, with the below images highlighting better reflection and rendering capabilities respectively.

In consultation with NVIDIA, we upgraded BenchBot to move from the old Unreal Engine-powered simulations to Omniverse. The upgrades bought technical challenges, improvements to BenchBot's robot platform integrations, and ultimately a deeper collaboration with NVIDIA's Embedded AI team. This collaboration led to me writing a technical blog post for NVIDIA, which was featured on the front page of their developers site.

Using BenchBot

The main BenchBot repository contains the software stack needed to develop solutions for BenchBot tasks on your local machine. It installs and configures a significant amount of software for you, wraps ~50GB of software in stable Docker images, and provides simple interaction with the stack through 4 basic scripts: benchbot_install, benchbot_run, benchbot_submit, and benchbot_eval.

The stack also handles installing our set of semantic scene understanding (SSU) add-ons by default. These add-ons can be swapped out, expanded on, or replaced with your own. Documentation for the add-ons system can be found in the benchbot_addons repository.

Where BenchBot is used

We've used BenchBot in a number of places already in our research, including our semantic scene understanding challenges. See below for an overview of how the challenges work.

The "bench" in "BenchBot" refers to benchmarking, with our goal to provide a system that greatly simplifies the benchmarking of novel algorithms in both realistic 3D simulation and on real robot platforms. If there is something else you would like to use BenchBot for (like integrating different simulators), please let us know. We're very interested in BenchBot being the glue between your novel robotics research and whatever your robot platform may be.