Peter Tong

Hi, I am Peter Tong, also go by the name Shengbang Tong(童晟邦). I am a first-year PhD student in NYU Courant CS advised by Professor Yann LeCun and Professor Saining Xie. I recently graduateed from UC Berkeley with a triple major in Computer Science, Applied Mathematics(Honor) and Statistic(Honor). I am from Nanjing, China and Melbourne, Australia.

Email  /  Resume  /  Twitter  /  Google Scholar  /  Github

I recently graduated from UC Berkeley with a triple major. I am a first-year CS PhD student in NYU Courant advised by Prof. Yann LeCun and Prof. Saining Xie. I was a researcher in Berkeley Artificial Intelligence Lab(BAIR) advised by Prof. Yi Ma and Prof.Jacob Steinhardt. I am interested in world model, unsupervised/self-supervised learning, generative models and multimodal models. I would like to thank all my mentors-Yubei, Xili, Erik and collaborators for the incredible journey I had in my undergrad.

• 2024-09: Our paper RLVLM was accepted at NeurIPS2024, and Cambrian was accepted at NeurIPS 2024 as an Oral Paper!
• 2024-05: I joined FAIR, Meta for summer internship with Dr. Zhuang Liu, yayyyyy!
• 2024-04: I received OpenAI Superalignment Fellowship! Thank you OpenAI!!! Looking forward to the cool works.
• 2024-04: Our paper Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs was accepted at CVPR 2024 as an Oral Paper!
• 2024-01: Our paper Image Clustering in the Age of Pretrained Models was accepted at ICLR 2024!
• 2023-12: Our papers were accepted at CPAL 2024!
• 2023-09: I am helping organizing the QVCV workshop in ICCV. See you all in Paris!
• 2023-09: Our papers MultiMon and CRATE(whitebox-transformer) were accepted at NeurIPS 2023!
• 2023-07: Our paper Manifold Linearizing and Clustering was accepted at ICCV 2023!
• 2023-05: I graduated from UC Berkeley with triple degree Applied Math (Honor), Statistics (Honor) and Computer Science (No honor, because I didn't want to take 16b and 61c too early, but I published quite some interesting work so yay)!!!
• 2023-04: I will be a CS PhD student in NYU Courant advised by Professor Yann LeCun and Professor Saining Xie. Looking forward to working with Yann and Saining in New York!
• 2023-01: Our paper incremental-CTRL was accepted at ICLR 2023!

We provide a vision-centric exploration or cookbook in MLLMs. In other words, we systematically study visual representation, vision-language connector, instruction tuning data, training recipe and evaluation protocols in MLLMs. We propose new vision-centric benchmarks, spatial-aware connector, data collection and curation of instruciton data, and more! We also release very competitive 8B, 13B and 34B models on par with GPT-4V and Gemini.

Is vision good enough for language? Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities.

We can use RL to train MLLM instead on SFT! Using RL to train from environment feedback unlocks model's ability in decision making, exceeding the limitations in SFT.

Deployed multimodal systems can fail in ways that evaluators did not anticipate. In order to find these failures before deployment, we introduce MULTIMON, a system that automatically identifies systematic failures.

Inspired by the newly proposed principle, our work proposes a minimalist method for self-supervised learning that tremendously reduces the epochs which SSL methods take to converge.

TLDR: Fine-Tuning multimodal large language models (MLLMs) leads to catastrophic forgetting.

Through extensive experimental results, we demonstrate that when employing a white-box transformer-like architecture known as CRATE, whose design explicitly models and pursues low-dimensional structures in the data distribution, segmentation properties, at both the whole and parts levels, already emerge with a minimalistic supervised training recipe. Layer-wise finer-grained analysis reveals that the emergent properties strongly corroborate the designed mathematical functions of the white-box network.

In this paper, we propose a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale. We show that the pre-trained features are significantly more structured by further optimizing the rate reduction objective.

In this paper, we contend that the objective of representation learning is to compress and transform the distribution of the data towards a mixture of low-dimensional Gaussian distributions supported on incoherent subspaces. From this perspective, popular deep networks such as transformers can be naturally viewed as realizing iterative schemes to optimize this objective incrementally.

This paper proposes a new unsupervised method to learn a structured representation that may serve both discriminative and generative purpose

This paper explores the natural inverse in Covolutional Sparse Coding neural network and its application in generative models.

This paper proposes a new unsupervised method to learn a represenation and cluster for real life dataset such as CIFAR-10, CIFAR100 and Tiny-ImageNet-200.

Our method uses differentiable optimization layers that are defined from convolutional sparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks. We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100 and ImageNet datasets when compared to conventional neural networks.

We propose a minimal computational model for learning a structured memory of multiple object classes in an incremental setting

We propose a new computational framework for learning an explicit generative model for real-world dataset.