메타학습을 이용한 시각-언어 모델 프롬프트 튜닝

초록

Recently, applying large pre-trained vision-language models such as CLIP to various downstream tasks has shown good performance. In low-shot image classification, simply optimizing continuous prompt vectors emerged, but has the limitation of low generalizability. Recent research introduces additional structures and algorithms to solve this problem, but there is a disadvantage of inefficiency. So, we propose a meta-training framework utilizing multi-modal features to optimize the prompt vectors. Our proposed method achieves over a 9.6% accuracy improvement compared to other models when evaluated on both seen and unseen domains comprehensively, which has no any additional memory overhead or inference latency for zero-shot inference.

키워드