Ben Talbot

Ruth Schulz

Ben Upcroft

Gordon Wyeth

Read the paper

View details in ACRA proceeedings

View details on QUT ePrints

ARC Discovery Project - DP140103216

QUT Centre for Robotics (QCR)

abstract-map-2015-acra

This paper presents a symbolic navigation system that uses spatial language descriptions to inform goal-directed exploration in unfamiliar office environments. An abstract map is created from a collection of natural language phrases describing the spatial layout of the environment. The spatial representation in the abstract map is controlled by a constraint based interpretation of each natural language phrase. In goal-directed exploration of an unseen office environment, the robot links the information in the abstract map to observed symbolic information and its grounded world representation.


Reasoning about Natural Language Phrases for Semantic Goal Driven Exploration

Obadiah Lam

Feras Dayoub

Peter Corke

View publication on IEEE

Watch a video of the paper results

abstract-map-2015-icra

In this paper we present for the first time a complete symbolic navigation system that performs goal-directed exploration to unfamiliar environments on a physical robot. We introduce a novel construct called the abstract map to link provided symbolic spatial information with observed symbolic information and actual places in the real world. Symbolic information is observed using a text recognition system that has been developed specifically for the application of reading door labels.


Robot navigation using human cues: A robot navigation system for symbolic goal-directed exploration

View details in workshop proceeedings

abstract-map-2015-rss

This paper describes ongoing work on a system using spatial descriptions to construct abstract maps that can be used for goal-directed exploration in an unfamiliar office environment. Abstract maps contain membership, connectivity, and spatial layout information extracted from symbolic spatial information. In goal-directed exploration, the robot would then link this information with observed symbolic information and its grounded world representation.


Constructing Abstract Maps from Spatial Descriptions for Goal-directed Exploration

abstract-map-2016-icra

This paper shows that by using only symbolic language phrases, a mobile robot can purposefully navigate to specified rooms in previously unexplored environments. The robot intelligently organises a symbolic language description of the unseen environment and “imagines” a representative map, called the abstract map. The abstract map is an internal representation of the topological structure and spatial layout of symbolically defined locations.


Find my office: Navigating real space from semantic descriptions

Read PhD thesis

abstract-map-2018-phd

Navigation cues – like labels, signs, maps, planners, spoken directions, and navigational gestures – provide humans with navigation capabilities that far surpass those of mobile robots. This thesis demonstrates how a mobile robot can use the symbolic spatial information embedded in navigation cues to proficiently navigate unseen built environments. The primary contributions of the thesis reside in the abstract map, a novel tool for grounding navigation symbols. The abstract map imagines abstract layouts and structures for unseen spaces through a mechanical analogy, then tethers the abstract spatial models with robot perceptions to inform navigation to symbolic goals in built environments.


Integrating Symbolic Spatial Information in Robot Navigation

View preprint on arXiv

Watch the robot navigate

View the code on GitHub

abstract-map-2020-tcds

Human navigation in built environments depends on symbolic spatial information which has unrealised potential to enhance robot navigation capabilities. Information sources such as labels, signs, maps, planners, spoken directions, and navigational gestures communicate a wealth of spatial information to the navigators of built environments; a wealth of information that robots typically ignore. We present a robot navigation system that uses the same symbolic spatial information employed by humans to purposefully navigate in unseen built environments with a level of performance comparable to humans.


Robot Navigation in Unseen Spaces using an Abstract Map

David Hall

Suman Bista

Haoyang Zhang

Rohan Smith

Niko Sünderhauf

View the dataset through QCR

View the publication on Sage

bear-2022-ijrr

We present a platform to foster research in active scene understanding, consisting of high-fidelity simulated environments and a simple yet powerful API that controls a mobile robot in simulation and reality. In contrast to static, pre-recorded datasets that focus on the perception aspect of scene understanding, agency is a top priority in our work. We provide three levels of robot agency, allowing users to control a robot at varying levels of difficulty and realism.


BenchBot environments for active robotics (BEAR): Simulated data for active scene understanding research

benchbot-2020-arxiv

We introduce BenchBot, a novel software suite for benchmarking the performance of robotics research across both photorealistic 3D simulations and real robot platforms. BenchBot provides a simple interface to the sensorimotor capabilities of a robot when solving robotics research problems; an interface that is consistent regardless of whether the target platform is simulated or a real robot. In this paper we outline the BenchBot system architecture, and explore the parallels between its user-centric design and an ideal research development process devoid of tangential robot engineering challenges. The paper describes the research benefits of using the BenchBot system, including: enhanced capacity to focus solely on research problems, direct quantitative feedback to inform research development, tools for deriving comprehensive performance characteristics, and submission formats which promote sharability and repeatability of research outcomes. BenchBot is publicly available ([here](http://benchbot.org)), and we encourage its use in the research community for comprehensively evaluating the simulated and real world performance of novel robotic algorithms.


BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots

Krishan Rana

Vibhavari Dasagi

Michael Milford

Read short research overview

Australian Centre for Robotic Vision (ACRV)

controller-fusion-2020-icra

In this work we focus on improving the efficiency and generalisation of learned navigation strategies when transferred from its training environment to previously unseen ones. We present an extension of the residual reinforcement learning framework from the robotic manipulation literature and adapt it to the vast and unstructured environments that mobile robots can operate in. The concept is based on learning a residual control effect to add to a typical sub-optimal classical controller in order to close the performance gap, whilst guiding the exploration process during training for improved data efficiency.


Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments

Watch a video summary of the research

controller-fusion-2020-iros

Learning-based approaches often outperform hand-coded algorithmic solutions for many problems in robotics. However, learning long-horizon tasks on real robot hardware can be intractable, and transferring a learned policy from simulation to reality is still extremely challenging. We present a novel approach to model-free reinforcement learning that can leverage existing sub-optimal solutions as an algorithmic prior during training and deployment.


Multiplicative Controller Fusion: Leveraging Algorithmic Priors for Sample-efficient Reinforcement Learning and Safe Sim-To-Real Transfer

Jesse Haviland

View in workshop proceedings

controller-fusion-2021-neurips

While deep reinforcement learning (RL) agents have demonstrated incredible potential in attaining dexterous behaviours for robotics, they tend to make errors when deployed in the real world due to mismatches between the training and execution environments. In contrast, the classical robotics community have developed a range of controllers that can safely operate across most states in the real world given their explicit derivation. These controllers however lack the dexterity required for complex tasks given limitations in analytical modelling and approximations. In this paper, we propose Bayesian Controller Fusion (BCF), a novel uncertainty-aware deployment strategy that combines the strengths of deep RL policies and traditional handcrafted controllers.


Zero-Shot Uncertainty-Aware Deployment of Simulation Trained Policies on Real-World Robots

controller-fusion-2022-ijrr

We present Bayesian Controller Fusion (BCF): a hybrid control strategy that combines the strengths of traditional hand-crafted controllers and model-free deep reinforcement learning (RL). BCF thrives in the robotics domain, where reliable but suboptimal control priors exist for many tasks, but RL from scratch remains unsafe and data-inefficient. By fusing uncertainty-aware distributional outputs from each system, BCF arbitrates control between them, exploiting their respective strengths. We study BCF on two real-world robotics tasks involving navigation in a vast and long-horizon environment, and a complex reaching task that involves manipulability maximisation.


Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Visit Emobided AI workshop site

embodied-ai-2022-arxiv

We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.


Retrospectives on the Embodied AI Workshop

Gavin Suddrey

Frederic Mair

View the behaviour trees project

lingua-2022-ral

Domestic and service robots have the potential to transform industries such as health care and small-scale manufacturing, as well as the homes in which we live. However, due to the overwhelming variety of tasks these robots will be expected to complete, providing generic out-of-the-box solutions that meet the needs of every possible user is clearly intractable. To address this problem, robots must therefore not only be capable of learning how to complete novel tasks at run-time, but the solutions to these tasks must also be informed by the needs of the user. In this paper we demonstrate how behaviour trees, a well established control architecture in the fields of gaming and robotics, can be used in conjunction with natural language instruction to provide a robust and modular control architecture for instructing autonomous agents to learn and perform novel complex tasks.


Learning and Executing Re-usable Behaviour Trees from Natural Language Instruction

Sourav Garg

openseqslam2-2018-iros

Visually recognising a traversed route - regardless of whether seen during the day or night, in clear or inclement conditions, or in summer or winter - is an important capability for navigating robots. Since SeqSLAM was introduced in 2012, a large body of work has followed exploring how robotic systems can use the algorithm to meet the challenges posed by navigation in changing environmental conditions. The following paper describes OpenSeqSLAM2.0, a fully open-source toolbox for visual place recognition under changing conditions.


OpenSeqSLAM2.0: An Open Source Toolbox for Visual Place Recognition under Changing Conditions

Sean McMahon

Watch videos of the system in action

place-categorisation-2016-icra

In this paper we focus on the challenging problem of place categorization and semantic mapping on a robot without environment-specific training. Motivated by their ongoing success in various visual recognition tasks, we build our system upon a state-of-the-art convolutional network. We overcome its closed-set limitations by complementing the network with a series of one-vs-all classifiers that can learn to recognize new semantic classes online. Prior domain knowledge is incorporated by embedding the classification system into a Bayesian filter framework that also ensures temporal coherence.


Place Categorization and Semantic Mapping on a Mobile Robot

semantic-slam-2021-iros

Recent Semantic SLAM methods combine classical geometry-based estimation with deep learning-based object detection or semantic segmentation. In this paper we evaluate the quality of semantic maps generated by state-of-the-art class-and instance-aware dense semantic SLAM algorithms whose codes are publicly available and explore the impacts both semantic segmentation and pose estimation have on the quality of semantic maps.


Evaluating the Impact of Semantic Segmentation and Pose Estimation on Dense Semantic SLAM

Research Publications