Chain of Verification (CoVe)¶

This notebook is a basic implementation of the paper Chain-of-Verification Reduces Hallucination In Large Language Models (2023) (arXiv: [2309.11495]).

In [1]:

# Define the prompts

import asyncio

from magentic import prompt


@prompt("{query}")
async def answer_query(query: str) -> str: ...


@prompt(
    """\
Query: {query}
Response: {response}

Provide specific questions to verify the facts in the above response as related to the query.
"""
)
async def generate_verification_questions(query: str, response: str) -> list[str]: ...


@prompt(
    """\
{context}

Given the above context, what is the answer to the following question?

{query}"""
)
async def answer_query_with_context(query: str, context: str) -> str: ...

# Define the prompts import asyncio from magentic import prompt @prompt("{query}") async def answer_query(query: str) -> str: ... @prompt( """\ Query: {query} Response: {response} Provide specific questions to verify the facts in the above response as related to the query. """ ) async def generate_verification_questions(query: str, response: str) -> list[str]: ... @prompt( """\ {context} Given the above context, what is the answer to the following question? {query}""" ) async def answer_query_with_context(query: str, context: str) -> str: ...

In [2]:

# 1. Generate Baseline Response
# Given a query, generate the response using the LLM.

query = "Name some politicians who were born in NY, New York"

baseline_response = await answer_query(query)
print(baseline_response)

# 1. Generate Baseline Response # Given a query, generate the response using the LLM. query = "Name some politicians who were born in NY, New York" baseline_response = await answer_query(query) print(baseline_response)

Here are some politicians who were born in New York, New York:

1. Franklin D. Roosevelt - 32nd President of the United States.
2. Theodore Roosevelt - 26th President of the United States.
3. Donald Trump - 45th President of the United States.
4. Hillary Clinton - Former Secretary of State and Democratic nominee for President in 2016.
5. Michael Bloomberg - Former Mayor of New York City and businessman.
6. Rudy Giuliani - Former Mayor of New York City and attorney.
7. Chuck Schumer - U.S. Senator from New York and current Senate Majority Leader.
8. Kirsten Gillibrand - U.S. Senator from New York.
9. Mario Cuomo - Former Governor of New York.
10. Andrew Cuomo - Current Governor of New York.

Please note that this is not an exhaustive list, and there are many more politicians who were born in New York, New York.

In [3]:

# 2. Plan Verifications
# Given both query and baseline response, generate a list of verification questions
# that could help to self-analyze if there are any mistakes in the original response.

verification_questions = await generate_verification_questions(query, baseline_response)

for q in verification_questions:
    print(q)

# 2. Plan Verifications # Given both query and baseline response, generate a list of verification questions # that could help to self-analyze if there are any mistakes in the original response. verification_questions = await generate_verification_questions(query, baseline_response) for q in verification_questions: print(q)

Was Franklin D. Roosevelt born in New York, New York?
Was Theodore Roosevelt born in New York, New York?
Was Donald Trump born in New York, New York?
Was Hillary Clinton born in New York, New York?
Was Michael Bloomberg born in New York, New York?
Was Rudy Giuliani born in New York, New York?
Was Chuck Schumer born in New York, New York?
Was Kirsten Gillibrand born in New York, New York?
Was Mario Cuomo born in New York, New York?
Is Andrew Cuomo the current Governor of New York?

In [4]:

# 3. Execute Verifications
# Answer each verification question in turn, and hence check the answer against the
# original response to check for inconsistencies or mistakes.

verification_answers = await asyncio.gather(
    *(answer_query(question) for question in verification_questions)
)

for ans in verification_answers:
    print(ans)

# 3. Execute Verifications # Answer each verification question in turn, and hence check the answer against the # original response to check for inconsistencies or mistakes. verification_answers = await asyncio.gather( *(answer_query(question) for question in verification_questions) ) for ans in verification_answers: print(ans)

Yes, Franklin D. Roosevelt was born on January 30, 1882, in Hyde Park, New York, which is located in Dutchess County.
Yes, Theodore Roosevelt was born in New York City, New York on October 27, 1858. Specifically, he was born in a house located at 28 East 20th Street in Manhattan.
Yes, Donald Trump was indeed born in New York, New York on June 14, 1946.
No, Hillary Clinton was not born in New York, New York. She was born on October 26, 1947, in Chicago, Illinois.
No, Michael Bloomberg was born in Boston, Massachusetts on February 14, 1942.
Yes, Rudy Giuliani was born in New York, New York. He was born on May 28, 1944, in the Brooklyn borough of New York City.
Yes, Chuck Schumer was born in Brooklyn, New York on November 23, 1950.
No, Kirsten Gillibrand was born in Albany, New York on December 9, 1966.
Yes, Mario Cuomo was born in New York City, New York, United States. He was born on June 15, 1932, in the borough of Queens, specifically in the neighborhood of South Jamaica.
As of September 2021, Andrew Cuomo is the current Governor of New York. However, please note that political positions can change, and it is always recommended to verify the information with up-to-date sources.

In [5]:

# 4. Generate Final Verified Response
# Given the discovered inconsistencies (if any), generate a revised response
# incorporating the verification results.

context = "\n".join(verification_answers)
verified_response = await answer_query_with_context(query, context)
print(verified_response)

# 4. Generate Final Verified Response # Given the discovered inconsistencies (if any), generate a revised response # incorporating the verification results. context = "\n".join(verification_answers) verified_response = await answer_query_with_context(query, context) print(verified_response)

Some politicians who were born in New York, New York include Franklin D. Roosevelt, Theodore Roosevelt, Donald Trump, Rudy Giuliani, and Mario Cuomo.