Updates | mFACT | Loss-weighting | Cite | Paper
Code and materials for the paper "Detecting and Mitigating Hallucinations in Multilingual Summarisation".
Please see the detailed instructions for using our mFACT metrics in mfact, and our loss weighing model in loss-weighting.
- mFACT metrics for six languages are now avaliable at HuggingFace! (20/05/2023)
Here is a quick navigation to all our released materials.
We upload our curated multilingual faithfulness classification dataset in google drive.
| Languages | mFACT |
| Chinese | HF link |
| Spanish | HF link |
| French | HF link |
| Hindi | HF link |
| Vietnamese | HF link |
| Turkish | HF link |
| English (mFACT-Transfer) | HF link |
Note: mFACT-Transfer - this is the version we trained on our English faithfulness classification dataset and performed zero-shot transfer for other languages.
Please check both our trained language and summarisation adapter in huggingface.
| Languages | R-1 | mFACT | mFACT-Transfer | Outputs |
| Chinese | 31.25 | 42.97 | 36.02 | Google drive |
| Spanish | 23.49 | 23.37 | 33.11 | Google drive |
| French | 27.46 | 37.11 | 40.88 | Google drive |
| Hindi | 24.97 | 34.26 | 26.46 | Google drive |
| Vietnamese | 28.04 | 39.47 | 38.20 | Google drive |
| Turkish | 17.38 | 37.80 | 29.20 | Google drive |
Please check both our trained language and summarisation adapter in huggingface.
| Languages | R-1 | mFACT | mFACT-Transfer | Outputs |
| Chinese | 28.97 | 34.58 | 30.62 | Google drive |
| Spanish | 22.83 | 21.24 | 29.28 | Google drive |
| French | 25.80 | 40.24 | 43.64 | Google drive |
| Hindi | 24.79 | 25.74 | 16.89 | Google drive |
| Vietnamese | 26.97 | 34.59 | 35.21 | Google drive |
| Turkish | 17.05 | 32.15 | 22.63 | Google drive |
@misc{qiu2023detecting,
title={Detecting and Mitigating Hallucinations in Multilingual Summarisation},
author={Yifu Qiu and Yftah Ziser and Anna Korhonen and Edoardo M. Ponti and Shay B. Cohen},
year={2023},
eprint={2305.13632},
archivePrefix={arXiv},
primaryClass={cs.CL}
}