awesome-personalized-lmms

Personalized Personal Assistant Models Awesome

📝 A curated list about Personalized Personal Assistant Models~ 📚

🌱 Contributing

Please feel free to create a pull request to add papers or edit any informations:

 
# Problem Settings: Using 3-5 images of a novel concept/subject (e.g., a pet named <bo>), can we personalize Large Multimodal Models so that:
(1) They retain their original capabilities (e.g., Describe a dog)
while (2) Enabling #tailored their capabilities for the novel concept? (e.g., Describe <bo>)

Table of Contents


* 🙋‍♀️ Personalization has been extensively explored in AI/ML/CV… It’s now time for personalizing Large Multimodal Models! 🙋‍♀️*		  
#  
<!– Over the years, we’ve witnessed the evolution of personalization across various tasks (e.g., object segmentation, image generation).
Now, with the rise of Large Multimodal Models (LMMs) – We have opportunities to personalizing these generalist, large-scale AI systems.
It’s time to take the leap and bring personalization into the realm of Large Multimodal Models, making them not only powerful but also user-specific!
^ Above caption are actually generated by GPT-4o, I feed it the figure and asked it to generate a caption, haha!  

(This figure is created by me. If there is anything incorrect, please feel free to correct me! Thank you!) –>

Blogs + Frameworks

Blogs:

Frameworks:

Papers

Personalized Asssistant
Title Venue Year Input Output Link/ Code
─── Robotics ───          
See, Act, Adapt: Active Perception for Unsupervised Cross-Domain Visual Adaptation via Personalized VLM-Guided Agent arXiv 2026 image, text text  
─── Agentic ───          
According to Me: Long-Term Personalized Referential Memory QA arXiv 2026 image, text text Code
PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory arXiv 2025 text text Data
─── Unified Models ───          
TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant KDD 2025 image, text image, text Code
UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens NeurIPS 2025 image, text image, text Page
YoChameleon: Personalized Vision and Language Generation CVPR 2025 image, text image, text Page
─── Vision Language Model ───          
PEARL: Personalized Streaming Video Understanding Model arXiv 2026 video, text text Code
Ego: Embedding-Guided Personalization of Vision-Language Models arXiv 2026 video, image, text text  
Contextualized Visual Personalization in Vision-Language Models arXiv 2026 image, text text Code
Online-PVLM: Advancing Personalized VLMs with Online Concept Learning arXiv 2025 image, text text  
MMPB: It’s Time for Multi-Modal Personalization NeurIPS 2025 image, text text Page
RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models NeurIPS 2025 image, text text Code
Training-Free Personalization via Retrieval and Reasoning on Fingerprints arXiv 2025 image, text text  
PVChat: Personalized Video Chat with One-Shot Learning arXiv 2025 video, text text  
Concept-as-Tree: Synthetic Data is All You Need for VLM Personalization arXiv 2025 image, text text  
Personalization Toolkit: Training Free Personalization of Large Vision Language Models arXiv 2025 image, text text  
Personalized Large Vision-Language Models arXiv 2024 image, text text  
MC-LLaVA: Multi-Concept Personalized Vision-Language Model arXiv 2024 image, text text Code
Personalized Visual Instruction Tuning ICLR 2025 image, text text  
Retrieval-Augmented Personalization for Multimodal Large Language Models CVPR 2025 image, text text Page, Code
MyVLM: Personalizing VLMs for user-specific queries ECCV 2024 image, text text Page, Code
Yo’LLaVA: Your Personalized Language and Vision Assistant NeurIPS 2024 image, text text Page, Code
─── Large Language Models ───          
Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval ICLR 2026 text text  
PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants ACL Findings 2025 text text Paper
PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory arXiv 2025 text text Data
Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale COLM 2025 text text  
Scaling Synthetic Data Creation with 1,000,000,000 Personas arXiv 2024 text text  
Personalized Large Language Models ICDMw 2024 text text  
LaMP: When Large Language Models Meet Personalization ACL 2024 text text Page, Code
Learning to Predict Persona Information forDialogue Personalization without Explicit Persona Description ACL 2023 text text  
Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge AAAI 2022 text text Code
A Personalized Dialogue Generator with Implicit User Persona Detection COLING 2022 text text  
Personalizing Dialogue Agents: I have a dog, do you have pets too? ACL 2018 text text  
Personalized Representation Learning / Personalized Image Retrieval

| Title | Venue | Year | Link/ Code | |:——– |:——–:|:—-:|:————-:| add https://arxiv.org/html/2603.01493v1 here? code is in https://github.com/LaVieEnRose365/PhotoBench | Personalized Representation from Personalized Generation | ICLR | 2025 | Code | | “This is my unicorn, Fluffy”: Personalizing frozen vision-language representations | ECCV | 2024 | Code |

Datasets

Name Year # Concepts Link Notes
ConCon-Chi 2024 20 GitHub with ConCon-Chi
PODS 2024 100 GitHub with personalized-rep
MC-LLaVA 2024 – GitHub with MC-LLaVA, multiple concepts
Yo’LLaVA 2024 40 GitHub with Yo’LLaVA, single concept
MyVLM 2024 29 GitHub with MyVLM, single concept

⣶⣶⣶⣶⣶⣖⣒⡄⠀⣶⡖⠲⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣤⠠⡄⠀⠀⠀⠀ ⠙⠛⣿⣿⣿⡟⠛⠃⢀⣿⣿⣆⣦⣴⠂⠤⠀⠀⠀⣠⣤⣴⣆⠠⢄⠀⠀⠀⣤⡤⢤⣤⣤⠤⢄⠀⠀⢻⣿⣦⡇⢀⣤⢤⠀ ⠀⢀⣿⣿⣿⡇⠀⠀⢸⣿⣿⣿⠛⣿⣷⣄⡇⠀⣼⣿⣿⡟⢿⣷⡄⣣⠀⢘⣿⣿⣿⠿⣿⣧⣈⡆⠀⢹⣿⣿⣷⣾⣧⣴⠀ ⠀⢰⣿⣿⣿⠀⠀⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⠙⠛⣻⣧⣾⣿⣿⡷⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⢸⣿⣿⣿⣿⣿⡇⠀ ⠀⢸⣿⣿⣿⠀⠀⠀⢸⣿⣿⡿⠀⣿⣿⣿⠃⠀⣰⣾⣿⡿⣿⣿⣿⣟⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⢸⣿⣿⣿⣿⡏⢇⠀ ⠀⣼⣿⣿⣿⠀⠀⠀⣸⣿⣿⣟⢠⣿⣿⣿⠀⠀⣿⣿⡟⣇⣾⣿⣿⣯⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⢼⣿⣿⣿⣿⣷⡈⡀ ⠀⠻⠿⠿⠟⠀⠀⠀⠻⠿⠿⠏⠸⣿⣿⣿⠀⠀⢿⣿⣿⣿⣿⣿⣿⡇⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⣿⣿⣿⡟⢻⣿⣧⣇ ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠀⠀⠉⠉⠀⠀⠀⠉⠉⠁⠀⠉⠉⠉⠀⠀⠘⠙⠋⠁⠈⠋⠛⠉ ⠀⠀⠀⠀⠀⠀⢀⣠⣤⡀⠀⢀⣀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣤⡤⠠⡄⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⢹⣿⣄⠱⣠⣿⣧⣴⠀⠀⣠⣤⣤⣀⣀⡀⠀⠀⢀⣤⠤⡀⢀⣠⡤⢄⠀⠈⣿⣿⣦⡇⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠈⢿⣿⣷⣿⣿⣿⡏⠀⣾⣿⣿⣿⣶⣄⡉⡄⠀⣿⣿⣤⣝⢸⣿⣦⣼⠀⠀⣿⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀⠀⢿⣿⣿⣿⠏⠀⠐⣿⣿⣿⠉⣿⣿⣷⡇⠀⣽⣿⣿⣯⢸⣿⣿⣿⠀⠀⢹⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⠀⠀⢠⣿⣿⣿⠀⣿⣿⣿⡇⠀⣻⣿⣿⡷⢸⣿⣿⣿⠀⠀⢸⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⠀⠀⠀⢿⣿⣿⣄⣿⣿⣿⠇⠀⢹⣿⣿⣿⣸⣿⣿⣿⠀⠀⢠⣽⣧⡄⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⠛⠋⠀⠀⠀⠈⠛⠛⠛⠛⠛⠉⠀⠀⠈⠛⠛⠛⠋⠛⠛⠋⠀⠀⠈⠛⠛⠁⠀⠀⠀⠀⠀⠀⠀

And good luck with your research! 🤗✨