Peter Tong

Research

I graduated from UC Berkeley with a triple major. I am a second-year CS PhD student in NYU Courant advised by Prof. Yann LeCun and Prof. Saining Xie. I was a researcher in Berkeley Artificial Intelligence Lab(BAIR) advised by Prof. Yi Ma and Prof. Jacob Steinhardt. I am interested in world model, unsupervised/self-supervised learning, generative models and multimodal models. I would like to thank all my mentors-Yubei, Xili, Erik and collaborators for the incredible journey I had in my undergrad.

News

• 2024-09: Our paper RLVLM was accepted at NeurIPS2024, and Cambrian was accepted at NeurIPS 2024 as an Oral Paper!
• 2024-05: I joined FAIR, Meta for summer internship with Dr. Zhuang Liu, yayyyyy!
• 2024-04: I received OpenAI Superalignment Fellowship! Thank you OpenAI!!! Looking forward to the cool works.
• 2024-04: Our paper Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs was accepted at CVPR 2024 as an Oral Paper!
• 2024-01: Our paper Image Clustering in the Age of Pretrained Models was accepted at ICLR 2024!
• 2023-12: Our papers were accepted at CPAL 2024!
• 2023-09: I am helping organizing the QVCV workshop in ICCV. See you all in Paris!
• 2023-09: Our papers MultiMon and CRATE(whitebox-transformer) were accepted at NeurIPS 2023!
• 2023-07: Our paper Manifold Linearizing and Clustering was accepted at ICCV 2023!
• 2023-05: I graduated from UC Berkeley with triple degree Applied Math (Honor), Statistics (Honor) and Computer Science (No honor, because I didn't want to take 16b and 61c too early, but I published quite some interesting work so yay)!!!
• 2023-04: I will be a CS PhD student in NYU Courant advised by Professor Yann LeCun and Professor Saining Xie. Looking forward to working with Yann and Saining in New York!
• 2023-01: Our paper incremental-CTRL was accepted at ICLR 2023!

Publications

MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu

Technical Report

Visual understanding and visual generation are mutually beneficial in unified models! But visual understanding data is much more effective than visual generation. Capabilities in LLM can also transfer to unified models such as implicit reasoning!

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Shengbang Tong*, Ellis Brown*, Penghao Wu*, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, Saining Xie

NeurIPS 2024 Oral

We provide a vision-centric exploration or cookbook in MLLMs, systematically studying visual representation, vision-language connector, instruction tuning data, training recipe and evaluation protocols. We propose new vision-centric benchmarks, spatial-aware connector, data collection and curation of instruction data, and release very competitive 8B, 13B and 34B models on par with GPT-4V and Gemini.

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, Saining Xie

CVPR 2024 Oral

Is vision good enough for language? Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. We identify 'CLIP-blind pairs' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark.

Mass-Producing Failures of Multimodal Systems with Language Models

Shengbang Tong*, Erik Jones*, Jacob Steinhardt

NeurIPS 2023

Deployed multimodal systems can fail in ways that evaluators did not anticipate. In order to find these failures before deployment, we introduce MULTIMON, a system that automatically identifies systematic failures.