Checking AI Answers

Where to start doubting

Checking every line of an answer is hard. Instead, pick the parts that are easy to get wrong and costly when wrong. General explanations and overall flow are usually fine, but the more specific the information, the more likely something invented has slipped in. If the answer contains any of the items below, mark them and check them one by one. The more smoothly an answer reads, the easier it is to glide past these, so make a habit of deliberately stopping on them.

• Numbers, statistics, ratios, amounts
• Dates, years, sequences
• Names of people, organisations, books, papers
• Direct quotations
• Rules such as laws, regulations and fees
• Links and references

First, does the source exist?

A source the AI supplies is itself something to check. Search for the book or paper title to see whether it exists, and confirm the author and year. A real title with different content is also common. If there is a link, open it and look for what the AI claimed on that actual page. If the link is dead or leads somewhere unrelated, treat the claim as unsupported and check from scratch using other material.

Go back to the original

The original is more accurate than anyone's retelling of it. For laws and rules, check the official guidance of the government body or the text of the law itself; for statistics, the release from the agency that produced them; for research, the paper or at least its abstract. Use news articles and blogs as signposts towards the original, and make the final call from the original. If no original can be found, accept the claim only as "reportedly", and say it is unconfirmed when passing it on.

Confirm in two different places

Once you have confirmed something in one place, see whether a source unconnected to the first says the same. Note that ten articles copied from the same original are effectively one source; if several pieces share identical wording, they probably trace back to one origin. When two different kinds of source agree, such as an official body and a specialist publication, you can trust it fairly well. If sources disagree, go with the more recent one closer to the original.

Redo the numbers yourself

For answers involving numbers, recalculating yourself is the surest check. Confirm the totals, make sure units have not been mixed, and ask what the denominator of a ratio is. Confusing percent with percentage points is a frequent error: an interest rate going from 2% to 3% is a rise of one percentage point but a 50% increase in relative terms. When comparing two texts or a before-and-after version, use a text comparison tool rather than your eyes, and you will miss fewer changes.

The limits of letting AI check itself

Asking "was that answer right?" does not verify anything. The reply comes from the same process, so it may confidently say yes, or "correct" a right answer into a wrong one. Some techniques are still useful: have it list the grounds for its answer one by one and check those yourself, have it write the strongest objection to its own answer, or ask it to flag the parts it is least sure of. Use these to find what to check, and do the final checking against outside sources.

Check twice when it is the answer you wanted

People tend to skip checking when they get the answer they hoped for. This is called confirmation bias. AI also tends to agree with whoever is asking, so if your question carries your opinion, the answer is likely to support it. The moment an answer matches your view perfectly is the moment it most needs checking. Ask the same question from the opposite direction as well, and compare the grounds behind both answers. Simply asking neutrally, "give me the arguments for and against this claim" instead of "give me the evidence that this is right", noticeably reduces one-sided answers.

How thoroughly to check

Not every answer needs the same scrutiny. Matching the level of checking to the cost of being wrong saves time while still preventing serious mistakes.

• Ideas and rewording: your own reading is enough
• Text others will see: check numbers, names and sources against originals
• Money, health or legal matters: check official guidance, then consult a professional
• Irreversible decisions: use AI answers as reference only