BenchBot Software Stack

Ben Talbot
14 May 2019
Last updated
26 May 2022

14 May 2019

First internal presentation of BenchBot concept to ACRV review panel, focusing only on real robots

15 November 2019

Internal alpha release of BenchBot as a bridge between simulation and real robots (v0.1.0)

20 March 2020

Public beta release for testing (v0.5.0)

9 June 2020

First mainstream release, ready for public use (v1.0.0)

3 August 2020

Preprint released on arXiv describing the BenchBot software suite

17 February 2021

Support for more research problems, simulators, and robot platforms, through a new modular architecture powered by add-ons (v2.0.0)

17 February 2021

Robotic Vision Scene Understanding (RVSU) challenge released, powered by BenchBot, as part of the Embodied AI workshop at CVPR 2021

14 February 2022

Robotic Vision Scene Understanding (RVSU) challenge released, powered by BenchBot, as part of the Embodied AI workshop at CVPR 2022

18 February 2022

First version released with support for NVIDIA's Omniverse-powered Isaac Sim (v2.2.0)

2 March 2022

BenchBot environments for active robotics (BEAR) published in IJRR

26 May 2022

Technical blog featuring BenchBot released on NVIDIA's developer website

The BenchBot simulation environment, powered by Nvidia Isaac

The BenchBot software stack is a collection of software packages that allow end users to control robots in real or simulated environments with a simple python API. It leverages the simple "observe, act, repeat" approach to robot problems prevalent in reinforcement learning communities (OpenAI Gym users will find the BenchBot API interface very similar).

The BenchBot Story

BenchBot was created as a tool to assist in the research challenges faced by the semantic scene understanding community; challenges including understanding a scene in simulation, transferring algorithms to real world systems, and meaningfully evaluating algorithm performance. We've since realised, these challenges don't just exist for semantic scene understanding, they're prevalent in a wide range of robotic problems.

BenchBot run across a range of photorealistic environments

This led us to create version 2 of BenchBot with a focus on allowing users to define their own functionality for BenchBot through add-ons. Want to integrate your own environments? Plug-in new robot platforms? Define new tasks? Share examples with others? Add evaluation measures? This all now possible with add-ons, and you don't have to do anything more than add some YAML and Python files defining your new content.

Shortly after the release of version 2, NVIDIA began publicising their Omniverse platform as a major step forward in the 3D simulation and digital design landscape. This included a major upgrade to their Isaac Sim robotics simulator. The upgrade brought significant improvements to simulation realism, with the below images highlighting better reflection and rendering capabilities respectively.

Improved reflections in BenchBot's BEAR environments with Omniverse-powered Isaac Sim

Improved rendering of BenchBot's BEAR environments with Omniverse-powered Isaac Sim

In consultation with NVIDIA, we upgraded BenchBot to move from the old Unreal Engine-powered simulations to Omniverse. The upgrades bought technical challenges, improvements to BenchBot's robot platform integrations, and ultimately a deeper collaboration with NVIDIA's Embedded AI team. This collaboration led to me writing a technical blog post for NVIDIA, which was featured on the front page of their developers site.

The BenchBot story, featured on homepage of the NVIDIA developers page

Using BenchBot

The main BenchBot repository contains the software stack needed to develop solutions for BenchBot tasks on your local machine. It installs and configures a significant amount of software for you, wraps ~50GB of software in stable Docker images, and provides simple interaction with the stack through 4 basic scripts: benchbot_install, benchbot_run, benchbot_submit, and benchbot_eval.

The stack also handles installing our set of semantic scene understanding (SSU) add-ons by default. These add-ons can be swapped out, expanded on, or replaced with your own. Documentation for the add-ons system can be found in the benchbot_addons repository.

Where BenchBot is used

We've used BenchBot in a number of places already in our research, including our semantic scene understanding challenges. See below for an overview of how the challenges work.

The "bench" in "BenchBot" refers to benchmarking, with our goal to provide a system that greatly simplifies the benchmarking of novel algorithms in both realistic 3D simulation and on real robot platforms. If there is something else you would like to use BenchBot for (like integrating different simulators), please let us know. We're very interested in BenchBot being the glue between your novel robotics research and whatever your robot platform may be.


David HallHaoyang ZhangSuman BistaRohan SmithFeras DayoubNiko Sünderhauf

Related Links

View the code on GitHubBenchBot blog for NVIDIAarXiv BenchBot paperIJRR BEAR paperGo to the tutorialEvalAI challengeOverview of the scene understanding project

© Ben Talbot. All rights reserved.