Researchers have recently proposed novel architectures that replace manual annotations with weak supervision from foundation models. It is however unclear what is the impact on the quality of the learned concepts. To answer this question, we put various models to the test, analyzing their learned concepts empirically using a selection of significant metrics.
- Clone the repository:
git clone https://github.com/debryu/CQA.git
- Move inside the folder:
cd CQA
- Create a new Conda environment:
conda create -n vlmcbm python=3.12
conda activate vlmcbm
- Install the CQA library
pip install -e .
- Customize your
config.pyaccordingly to your preferences. Set the correct paths if you are not using the default ones.
First move into the CQA folder
cd CQA
I.E. when running training/evaluations you need to be in this path ./CQA/CQA.
To train a simple Concept Bottleneck Model use:
python train.py -model resnetcbm -dataset celeba -epochs 20 -unfreeze 5
To train using the CUB pre-trained backbone:
python train.py -model resnetcbm -backbone resnet18_cub -dataset celeba -epochs 20 -unfreeze 5
Commands to train different models are available in experiments
To evaluate a model run:
python main.py -load_dir <YOUR MODEL FOLDER> -all
- Label-free CBM: https://github.com/Trustworthy-ML-Lab/Label-free-CBM
- LaBo CBM: https://github.com/YueYANG1996/LaBo
- VLG-CBM: https://github.com/Trustworthy-ML-Lab/VLG-CBM
- CUB dataset: https://www.vision.caltech.edu/datasets/cub_200_2011/
- SHAPES3D dataset: https://github.com/google-deepmind/3d-shapes
- CELEBA dataset: https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
- DCI metrics: https://github.com/andreinicolicioiu/DCI-ES
- OIS metric: https://github.com/mateoespinosa/concept-quality
- Sparse final layer training: https://github.com/MadryLab/glm_saga
- CLIP: https://github.com/openai/CLIP
- Grounding DINO: https://github.com/IDEA-Research/GroundingDINO
- LLava Phi 3 (using Ollama): https://ollama.com/library/llava-phi3
@misc{debole2025conceptbottlenecksquestionfoundation,
title={If Concept Bottlenecks are the Question, are Foundation Models the Answer?},
author={Nicola Debole and Pietro Barbiero and Francesco Giannini and Andrea Passerini and Stefano Teso and Emanuele Marconato},
year={2025},
eprint={2504.19774},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2504.19774},
}