Multimodal Model Embodied Simulation

I was in the FMP research scholar program and researched on Affordance Embodied Simulation's presences in Multimodal Models through the UCSD Faculty Mentorship Program (Sep 2023 - Jun 2024) under the supervision of Sean Trott. We try to improve the reliability of machine learning models through examining the degree of models’ “understanding” of the subtle keys in human languages and how it is used to map the world we know.

The prelimiary goal for this project alighs with the idea of LLM-ology, or the study of Large Language Models (proposed by my advisor of this project, Sean Trott), hoping to develop more understanding of our language semantic system through reverse engineering it using Multimodal Models.

Demo of GPT clarifying meaning of words (borrowed from Sean's blog).

The FMP project was finished with the 2023 FMP Sympossium using this manuscript. The newer manuscript is submitted to ACL and is under review.