Human-like object concept representations emerge naturally in multimodal large language models
We collected 4.7 million triplet judgments from LLMs and Multimodal LLMs (MLLMs) to derive low-dimensional embeddings that capture the similarity structure of 1,854 natural objects. The resulting 66-dimensional embeddings were stable, predictive, and exhibited semantic clustering similar to human mental representations. Remarkably, the dimensions underlying these embeddings were interpretable, suggesting that LLMs and MLLMs develop human-like conceptual representations of objects. Further analysis showed strong alignment between model embeddings and neural activity patterns in brain regions such as EBA, PPA, RSC, and FFA. This provides compelling evidence that the object representations in LLMs, while not identical to human ones, share fundamental similarities that reflect key aspects of human conceptual knowledge. Our findings advance the understanding of machine intelligence and inform the development of more human-like artificial cognitive systems.
And:
In contrast, the purely vision model #SimCLR (a self-supervised learning model) shows minimal ability to learn semantic dimensions...
It will be intriguing to see results of similar research done with V-JEPA and DINO.
V-JEPA "should" show even close cognitive aligments, at least this is what I would expect: V-JEPA > DINO >> SimCLR.
Even though #SimCLR's underlying contrastive paradigm seems conceptually quite similar to the "Odd-One-Out" triplet task used in the article (which inherently relies on pushing an anomaly away and pulling a similar pair together), SimCLR "unnaturally", forcefully learns how to push / pull the pixel instance embeddings. irony
#DINO captures objects boundaries in more "organic" way through self-distillation.
However, #V-JEPA utilizes crucial spatio-temporal continiuty omitting pixel space reconstructure, operates on abstract level latent predictions instead.