This repository is the official PyTorch implementation of the ICLR 2025 paper: Attribute-based Visual Reprogramming for Image Classification with CLIP, authored by Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu.
Abstract: Visual reprogramming (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose Attribute-based Visual Reprogramming (AttrVR) for CLIP, utilizing descriptive attributes (DesAttrs) and distinctive attributes (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the k-nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models.
- Python (3.10.0)
- PyTorch (2.0.1)
- TorchVision (0.15.2)
conda create -n reprogram
conda activate reprogram
pip install -r requirement.txt
To implement the results, please follow CoOp to download the datasets and modify DOWNSTREAM_PATH = "" in cfg.py of this repository.
-
We have uploaded all generated attributes used in this paper to
attributes/gpt3. Other attributes used in Appendix C.7 & C.9 generated by other LLMs/MLLMs can also be found inattributes/.... -
If you would like to generate the attributes by yourself, please first enter your API Key in
generate_attributes.py, then run the codepython generated_attributes.py.
python experiments/fs_vp.py --dataset [dataset]
python experiments/fs_ar.py --dataset [dataset]
python experiments/fs_attrvr.py --dataset [dataset]
This repo is built upon these previous works:
@inproceedings{cai2025attribute,
title={Attribute-based Visual Reprogramming for Image Classification with CLIP},
author={Chengyi Cai and Zesheng Ye and Lei Feng and Jianzhong Qi and Feng Liu},
booktitle = {International Conference on Learning Representations},
year={2025}
}