HyperCLIP

HyperCLIP is a contrastive vision-language model that uses a hypernetwork to generate normalization parameters for a small image encoder from text-prompt embeddings. This repository contains the training and evaluation code accompanying the TMLR paper.

The implementation builds on OpenCLIP and DataComp.

Installation

The experiments were run with Python 3.10 and CUDA. Create the recorded Conda environment with:

conda env create -f environment.yml
conda activate datacomp

requirements.lock.txt provides a fully pinned package snapshot for reproducible installations.

Data

Training data must be provided as WebDataset .tar shards containing paired images and captions:

/path/to/train/
├── 00000000.tar
├── 00000001.tar
└── ...

Download the evaluation datasets into a local directory with:

python download_evalsets.py /path/to/eval-data

Large datasets and model checkpoints are not included in this repository.

Training

Training recipes are defined in scale_configs.py. The primary EfficientNet-B0 configurations are:

• rep_b0_siglip: matched SigLIP baseline
• rep_b0_hyperclip: HyperCLIP
• rep_b0_token32: prompt-independent capacity control
• rep_b0_linear32: linear-weight prediction control
• rep_b0_conv32: convolution-weight prediction control

For example, train HyperCLIP-B0 on four GPUs with:

torchrun --nproc_per_node 4 train.py \
  --scale rep_b0_hyperclip \
  --data_dir '/path/to/train/{00000000..00012556}.tar' \
  --output_dir output \
  --exp_name rep_b0_hyperclip-seed42 \
  --precision amp \
  --seed 42 \
  --dataset_resampled \
  --siglip

The main paper recipe uses a global batch size of 1500, 128 million training samples, a learning rate of 5e-4, and 500 warmup steps. These values are supplied by the selected training configuration. Run python train.py --help for all available options.

Evaluation

Prepare the evaluation datasets, then evaluate a training output directory with:

python evaluate.py \
  --train_output_dir output/rep_b0_hyperclip-seed42 \
  --data_dir /path/to/eval-data \
  --all_tasks

Evaluation metrics are written locally to eval_results.jsonl. The evaluator does not upload checkpoints or submit results to external services.

To evaluate a checkpoint only on ImageNet-1K and CIFAR-100:

python experiments/eval_in1k_c100.py \
  --arch HypTimEfNetb0 \
  --ckpt /path/to/epoch_latest.pt \
  --data_root /path/to/eval-data \
  --out output/b0_metrics.json

Camera-ready control experiments

The following scripts reproduce the controls reported in the camera-ready paper:

• experiments/e1_prompt_controls.py: prompt perturbations and prompt-count sweep
• experiments/e3_multiseed.sh: multi-seed EfficientNet-B0 comparison
• experiments/e5_caption_stats.py: caption and class-prompt embedding statistics
• experiments/e6_posthoc.py: post-hoc hypernetwork fit on a frozen SigLIP encoder
• experiments/check_hypernet_collapse.py: hypernetwork output-invariance diagnostic

Each Python script provides its required arguments through --help. For example:

python experiments/e1_prompt_controls.py \
  --ckpt /path/to/hyperclip_checkpoint.pt \
  --data_root /path/to/eval-data \
  --out output/prompt_controls.json

The multi-seed runner reads its paths from environment variables:

DATA='/path/to/train/{00000000..00012556}.tar' \
EVAL_DATA=/path/to/eval-data \
OUT=output \
bash experiments/e3_multiseed.sh

Repository structure

.
├── open_clip/       # HyperCLIP and CLIP models, losses, tokenizer, and configs
├── training/        # Training loop, distributed utilities, and data loading
├── eval_utils/      # Classification, retrieval, fairness, and shift evaluation
├── datasets/        # Dataset builders, class names, and prompt templates
├── experiments/     # Camera-ready control experiments
├── train.py         # Training entry point
├── evaluate.py      # Local evaluation entry point
└── scale_configs.py # Paper training recipes

License

The code is released under the Apache License 2.0. See LICENSE.