UCLA · COMPUTER SCIENCE · PHD

Cheng-Fu Yang 楊晟甫

Building multimodal agents that perceive, reason, and interact with the world.

I'm a fifth-year Ph.D. student at UCLA, advised by Prof. Kai-Wei Chang. My research focuses on grounded multimodal reasoning, agentic tool use, and representation learning for perception and planning.

Portrait of Cheng-Fu Yang
UCLA NLP · Los Angeles

01 / ABOUT

Research interests

My work explores how models can move beyond answering from a single observation: acquiring external evidence, combining different forms of information, and using that evidence to plan and act.

01

Multimodal agents
Grounded perception, reasoning, tool use, and interaction.

02

Representation learning
Visual understanding, planning, and embodied intelligence.

Exploring now: coding agents for 3D scene reconstruction and spatial understanding from real-world observations (ongoing research).

02 / SELECTED WORK

Research highlights

Recent work across multimodal agents, audiovisual perception, and interpretable representations.

LATEST · 2026Preprint · Under submission

OpenVLHarness

Real World Visual Agentic Tool-Use with Multimodal Memory

We build a general multimodal reasoning harness with task-level perception tools and persistent multimodal memory. It lets models gather visual evidence, inspect artifacts, and use code across reasoning steps—without additional model training.

22 benchmarks · +9.7–12.6 points across model backbones

CVPR 2026 · AUDIOVISUAL PERCEPTION

PE-AV

Large-scale multimodal correspondence learning for joint audio, video, and text representations.

Read paper ↗

ICCV 2025 · REPRESENTATION LEARNING

Verbalized Representation Learning

Human-interpretable, language-based visual features for few-shot generalization.

ICML 2026 · SPOTLIGHT

Ctrl-R

Tractable trajectory control for discovering and reinforcing structured reasoning patterns.

Read paper ↗

03 / PUBLICATIONS

Selected & recent publications

Complete list below. My name is highlighted. Google Scholar ↗

2026

Preprint · 2026 (under submission)

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Yu Zhou*, Cheng-Fu Yang*, Rui Sun*, Di Wu*, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang

ICML 2026 · Spotlight

Learning Structured Reasoning via Tractable Trajectory Control

Po-Nien Kung, Zhen Yang, Jeffrey Luo, Cheng-Fu Yang, Haikang Deng, Zi-Yi Dou, Yinfei Yang, Nanyun Peng, Zhe Gan, Kai-Wei Chang

CVPR 2026

Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning

Apoorv Vyas*, Heng-Jui Chang*, Cheng-Fu Yang*, Po-Yao Huang*, Luya Gao*, Julius Richter*, Sanyuan Chen, Matthew Le, Piotr Dollár, Christoph Feichtenhofer, Ann Lee, Wei-Ning Hsu

2025

COLM 2025

Customize Multi-modal RAI Guardrails with Precedent-based Predictions

Cheng-Fu Yang, Christos Christodoulopoulos, Thanh Tran, Weitong Ruan, Rahul Gupta, Kai-Wei Chang

ICCV 2025

Verbalized Representation Learning for Interpretable Few-Shot Generalization

Cheng-Fu Yang, Da Yin, Wenbo Hu, Heng Ji, Nanyun Peng, Bolei Zhou, Kai-Wei Chang

2024

NeurIPS OWA Workshop 2024

Planning as Inpainting: A Generative Framework for Realistic Embodied Path Planning

Cheng-Fu Yang, Haoyang Xu, Te-Lin Wu, Xiaofeng Gao, Kai-Wei Chang, Feng Gao

EMNLP 2024

LLM-A*: Large Language Model Enhanced Incremental Heuristic Search on Path Planning

Silin Meng, Yiwei Wang, Cheng-Fu Yang, Nanyun Peng, Kai-Wei Chang

EMNLP 2024

Re-ReST: Reflection-Reinforced Self-Training for Language Agents

Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, Nanyun Peng

2023

EMNLP 2023

LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following

Cheng-Fu Yang, Yen-Chun Chen, Jianwei Yang, Xiyang Dai, Lu Yuan, Yu-Chiang Frank Wang, Kai-Wei Chang

AAAI 2023

Target-free Text-guided Image Manipulation

Wan-Cyuan Fan, Cheng-Fu Yang, Chiao-An Yang, Yu-Chiang Frank Wang

2022

NeurIPS 2022

Paraphrasing Is All You Need for Novel Object Captioning

Cheng-Fu Yang, Yao-Hung Hubert Tsai, Wan-Cyuan Fan, Ruslan Salakhutdinov, Louis-Philippe Morency, Yu-Chiang Frank Wang

AAAI 2022

Cross-Modal Mutual Learning for Audio-Visual Speech Recognition and Manipulation

Chih-Chun Yang, Cheng-Fu Yang, Wan-Cyuan Fan, Yu-Chiang Frank Wang

CVPR 2022

Scene Graph Expansion for Semantics-Guided Image Outpainting

Chiao-An Yang, Cheng-Yo Tan, Wan-Cyuan Fan, Cheng-Fu Yang, Meng-Lin Wu, Yu-Chiang Frank Wang

2021

CVPR 2021

LayoutTransformer: Scene Layout Generation with Conceptual and Spatial Diversity

Cheng-Fu Yang*, Wan-Cyuan Fan*, Fu-En Yang, Yu-Chiang Frank Wang

* Contribution markers follow the corresponding papers. For official author contributions, refer to each publication.

04 / EXPERIENCE

Education & research experience

Education

2022 — Present

University of California, Los Angeles

Ph.D. in Computer Science

Advisor: Kai-Wei Chang

2022 — 2025

University of California, Los Angeles

M.S. in Computer Science

2016 — 2020

National Taiwan University

B.S. in Electrical Engineering

Industry & research

2026

Meta FAIR · Research Scientist Intern

Multimodal post-training / next-generation SAM

Communication & Language · Daniel Bolya, Chay Ryali

2025 — 2026

Meta FAIR · Research Scientist Intern

Large-scale omni-modal pre-training / PE-AV

Perception · Bernie Huang

2024

Amazon AGI · Applied Scientist Intern

Constituting policies for language agents

2023

Amazon · Applied Scientist Intern

Diffusion policies for embodied agents

2022 — 2023

Microsoft Research · Research Intern

Representation learning for instruction following

2020 — 2022

National Taiwan University · Research Assistant

Vision & Learning Lab