🔎 Planning as Inpainting:
A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty

Cheng-Fu Yang1   Haoyang Xu1   Te-Lin Wu1   Xiaofeng Gao2   Kai-Wei Chang1   Feng Gao2
1UCLA   2Amazon

arXiv Code BibTex

Abstract

Task planning for embodied AI has been one of the most challenging problems where the community does not meet a consensus in terms of formulation. In this paper, we aim to tackle this problem with a unified framework consisting of an end-to-end trainable method and a planning algorithm. Particularly, we propose a task-agnostic method named 'planning as in-painting'. In this method, we use a Denoising Diffusion Model (DDM) for plan generation, conditioned on both language instructions and perceptual inputs under partially observable environments. Partial observation often leads to the model hallucinating the planning. Therefore, our diffusion-based method jointly models both state trajectory and goal estimation to improve the reliability of the generated plan, given the limited available information at each step. To better leverage newly discovered information along the plan execution for a higher success rate, we propose an on-the-fly planning algorithm to collaborate with the diffusion-based planner. The proposed framework achieves promising performances in various embodied AI tasks, including vision-language navigation, object manipulation, and task planning in a photorealistic virtual environment.


Approach

overview

The figure above illustrates the proposed framework, integrating the diffusion-based planner into the on-the-fly algorithm.


Object Manipulation

Object Manipulation Video Comming Soon

Vision-language navigation

Vision-language navigation Video Comming Soon

Multi-Object Single Goal (MOG1) Results

MOG1

Comparative Analysis: Our model demonstrates a high fidelity in replicating the ground-truth paths against baseline models such as BC and CQL.


Partially-Observable Multi-Object Goal (P-MOG2) Results

MOG2

Comparative Analysis: Our model successfully navigates first to a visible reference object and then to an initially unseen target object, outperforming baseline models like BC and CQL that often take indirect routes or fail to reach the target.


BibTeX

@article{yang2023planning,
        title={Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty}, 
        author={Cheng-Fu Yang and Haoyang Xu and Te-Lin Wu and Xiaofeng Gao and Kai-Wei Chang and Feng Gao},
        year={2023},
        eprint={2312.01097},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
  }

Template from Nerfies