Measuring Attribution in Natural Language Generation Models

Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, David Reitter

Correct Metadata for

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.

Anthology ID:: 2023.cl-4.2
Volume:: Computational Linguistics, Volume 49, Issue 4 - December 2023
Month:: December
Year:: 2023
Address:: Cambridge, MA
Venue:: CL
SIG:
Publisher:: MIT Press
Note:
Pages:: 777–840
Language:
URL:: https://aclanthology.org/2023.cl-4.2/
DOI:: 10.1162/coli_a_00486
Bibkey:
Cite (ACL):: Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring Attribution in Natural Language Generation Models. Computational Linguistics, 49(4):777–840.
Cite (Informal):: Measuring Attribution in Natural Language Generation Models (Rashkin et al., CL 2023)
Copy Citation:
PDF:: https://aclanthology.org/2023.cl-4.2.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@article{rashkin-etal-2023-measuring,
    title = "Measuring Attribution in Natural Language Generation Models",
    author = "Rashkin, Hannah  and
      Nikolaev, Vitaly  and
      Lamm, Matthew  and
      Aroyo, Lora  and
      Collins, Michael  and
      Das, Dipanjan  and
      Petrov, Slav  and
      Tomar, Gaurav Singh  and
      Turc, Iulia  and
      Reitter, David",
    journal = "Computational Linguistics",
    volume = "49",
    number = "4",
    month = dec,
    year = "2023",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2023.cl-4.2/",
    doi = "10.1162/coli_a_00486",
    pages = "777--840",
    abstract = "Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="rashkin-etal-2023-measuring">
    <titleInfo>
        <title>Measuring Attribution in Natural Language Generation Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hannah</namePart>
        <namePart type="family">Rashkin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vitaly</namePart>
        <namePart type="family">Nikolaev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Matthew</namePart>
        <namePart type="family">Lamm</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lora</namePart>
        <namePart type="family">Aroyo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="family">Collins</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dipanjan</namePart>
        <namePart type="family">Das</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Slav</namePart>
        <namePart type="family">Petrov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gaurav</namePart>
        <namePart type="given">Singh</namePart>
        <namePart type="family">Tomar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Iulia</namePart>
        <namePart type="family">Turc</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Reitter</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.</abstract>
    <identifier type="citekey">rashkin-etal-2023-measuring</identifier>
    <identifier type="doi">10.1162/coli_a_00486</identifier>
    <location>
        <url>https://aclanthology.org/2023.cl-4.2/</url>
    </location>
    <part>
        <date>2023-12</date>
        <detail type="volume"><number>49</number></detail>
        <detail type="issue"><number>4</number></detail>
        <extent unit="page">
            <start>777</start>
            <end>840</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Journal Article
%T Measuring Attribution in Natural Language Generation Models
%A Rashkin, Hannah
%A Nikolaev, Vitaly
%A Lamm, Matthew
%A Aroyo, Lora
%A Collins, Michael
%A Das, Dipanjan
%A Petrov, Slav
%A Tomar, Gaurav Singh
%A Turc, Iulia
%A Reitter, David
%J Computational Linguistics
%D 2023
%8 December
%V 49
%N 4
%I MIT Press
%C Cambridge, MA
%F rashkin-etal-2023-measuring
%X Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.
%R 10.1162/coli_a_00486
%U https://aclanthology.org/2023.cl-4.2/
%U https://doi.org/10.1162/coli_a_00486
%P 777-840

Download as File

Markdown (Informal)

[Measuring Attribution in Natural Language Generation Models](https://aclanthology.org/2023.cl-4.2/) (Rashkin et al., CL 2023)

Measuring Attribution in Natural Language Generation Models (Rashkin et al., CL 2023)

ACL

Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring Attribution in Natural Language Generation Models. Computational Linguistics, 49(4):777–840.