Does Claude make up figures? Memory, web and the BCRP’s MCP

Updated:

AI evaluation · MCP · BCRP

Stacked bar chart of Claude Haiku's correct answers, errors and “doesn't know” answers on 10 BCRP questions, depending on whether it uses memory, web or MCP
Figure. Correct answers, errors and “doesn't know” answers from Claude Haiku on 10 BCRP questions, depending on whether it uses memory, web or MCP.

10/10 correct answers with the BCRP MCP server (web: 9, memory: 0)

Question

Does Claude answer official BCRP data correctly depending on where it gets the information?

Finding

With 10 questions and Claude Haiku, the MCP server got 10 right, the web 9 and memory 0 (it said “I don’t know”). When memory was denied the “I don’t know” option, 7 of 10 answers came out wrong: 3 were real estimates and 4 a zero used as a placeholder.

Report

Results by query mode

Correct Wrong Doesn't know

  • Memory base 10
  • Memory no null 7 3
  • Web base 9 1
  • MCP base 10
  • MCP no null 10

Hover over or focus a row to see its detail. “No null” takes away Claude's “I don't know” option.

Sources

The figures come from the project’s report and repository.

← All projects