This repo contains the code of our EMNLP 2022 paper, ROSE: Robust Selective Fine-tuning for Pre-trained Language Models.
We present an attack-agnostic and model-agnostic defense method called RObust SEletive fine-tuning (ROSE). ROSE conducts selective updates when adapting pre-trained models to downstream tasks, filtering out invaluable and unrobust updates of parameters. The experimental results show that ROSE achieves significant improvements in adversarial robustness on various downstream NLP tasks. Furthermore, ROSE can be easily incorporated into existing fine-tuning methods to improve their adversarial robustness further. ROSE eliminates unrobust spurious updates during fine-tuning, leading to solutions corresponding to flatter and wider optima than the conventional method. The following figure is an illustration of our models.
Install dependencies and apex:
conda create -n rose python=3.8
conda activate rose
git clone https://github.com/jiangllan/ROSE.git && cd ROSE
pip intall -r requirment.txt
git clone https://github.com/NVIDIA/apex
cd apex && pip install -v --no-cache-dir ./Please refer to runs for training and test commands.
- Training with ROSE-First on QNLI
# [model name] [n_gpu] [lr] [bs] [seed] [upper_thd] [lower_thd] [dropout] [task name]
bash run_train_sparse_first.sh roberta-base 1 0.00002 32 8910 70 0 0.1 qnliFor each task, we report the specific settings of several important hyper-parameters:
| Hyper-parameter | QNLI | QQP | RTE | SST-2 | QNLI | QQP | RTE | SST-2 |
|---|---|---|---|---|---|---|---|---|
--lr |
2e-5 | 2e-5 | 4e-5 | 2e-5 | 3e-5 | 2e-5 | 1e-5 | 4e-5 |
--upper |
70 | 70 | 80 | 60 | 50 | 70 | 80 | 70 |
- Training with ROSE-Second on QNLI
# [model name] [n_gpu] [lr] [bs] [seed] [upper_thd] [lower_thd] [task name]
bash run_train_sparse_second.sh roberta-base 1 0.00001 32 8910 40 0 qnliFor each task, we report the specific settings of several important hyper-parameters:
| Hyper-parameter | QNLI | QQP | RTE | SST-2 | QNLI | QQP | RTE | SST-2 |
|---|---|---|---|---|---|---|---|---|
--lr |
1e-5 | 1e-4 | 4e-5 | 1e-4 | 1e-5 | 1e-5 | 2e-5 | 1e-5 |
--upper |
40 | 60 | 60 | 30 | 50 | 60 | 50 | 50 |
- Training with ROSE-Ensemble on QNLI
# [model name] [n_gpu] [lr] [bs] [seed] [upper_thd] [lower_thd] [dropout] [task name] [gamma]
bash run_train_sparse_ensemble.sh roberta-base 1 0.00001 32 8910 90 0 0.1 qnli 0.5For each task, we report the specific settings of several important hyper-parameters:
| Hyper-parameter | QNLI | QQP | RTE | SST-2 | QNLI | QQP | RTE | SST-2 |
|---|---|---|---|---|---|---|---|---|
--lr |
1e-5 | 5e-5 | 4e-5 | 5e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 |
--upper |
90 | 75 | 80 | 40 | 30 | 80 | 85 | 70 |
- Evaluation on GLUE
# [path to checkpoint] [task name]
bash run_test_ori.sh ../temp/.. qnli- Evaluation on AdvGLUE
# [path to checkpoint] [task name]
bash run_test_adv.sh ../temp/.. qnlijiangl20 at mails dot tsinghua dot edu dot cn
This codebase is built on Huggingface's Transformers. Thanks to them!