Why GPT and Claude Failed the World's Biggest Hedge Fund's Finance Test — and a Custom Model Hit 84.7%

Bridgewater and Thinking Machines Lab tested Gemini, Claude, and GPT on the document-filtering work real investors do every day. The best frontier model topped out at 78.2% — below the 80% investors needed to trust it. A smaller custom model, trained on private expert answers, hit 84.7%. Here's why the gap matters more than any leaderboard.

If you only read AI benchmark headlines, you'd think frontier models had already conquered finance. They pass the CFA exam, ace math olympiad problems, and crush public reasoning tests. So when the world's largest hedge fund put GPT, Claude, and Gemini through a real-world finance test and none of them cleared the bar, it wasn't a story about dumb models. It was a story about what public benchmarks quietly leave out.

This post breaks down the actual numbers Bridgewater's AIA Labs and Thinking Machines Lab published, why "the answers were never public" is the whole point, and what it means if you're betting money — literally or in the market — on AI automating knowledge work.

What Bridgewater actually tested

The experiment focused on something unglamorous but central to how a hedge fund runs: document relevance filtering. Investors at Bridgewater sift through filings, news, and research all day, deciding which documents actually matter to a given question. That judgment call — is this relevant or not? — was the test.

Researchers ran six of these real document-filtering tasks against the leading frontier models. The results came in three tiers:

Two things jump out. First, a plain prompt got you nowhere — around 50%, the same as guessing. Second, even heavy prompt engineering by experts only lifted the best frontier model to 78.2%, still under the 80% threshold Bridgewater's investors said they'd need before trusting the system in their workflow. The only setup that crossed the line was a smaller, purpose-trained model fed the fund's own expert-labeled examples.

Why "the answers were never public" is the whole story

Here's the uncomfortable insight for the AI hype cycle. Frontier models are extraordinary at questions whose answers exist somewhere in their training data — public exams, textbook problems, coding puzzles that have been solved a thousand times on the open web. That's also exactly what most public benchmarks measure.

Bridgewater's relevance decisions don't live on the open web. The "right answer" for whether a document matters depends on the fund's private research process, its internal frameworks, and context no scraper ever saw. There was nothing to memorize. When you strip out the possibility of pattern-matching to something already seen, the models had to reason from scratch about a specialist's judgment — and that's where they fell short.

This is the practical face of benchmark contamination and overfitting. A model can look near-perfect on a public leaderboard and still be mediocre at your actual job, because the leaderboard rewards recall of public knowledge while your job requires private judgment. The 78.2% number isn't a knock on the models' raw intelligence; it's a measurement of how much of "intelligence" on benchmarks was quietly leaning on exposure.

The custom model beat the giants — and that's the takeaway for builders

The headline result is that a fine-tuned model trained on Bridgewater's proprietary, expert-labeled data reached 84.7%, beating the strongest general-purpose frontier model's 78.2% by more than six points. Reports around the research also noted the custom model was far smaller and cheaper to run than the frontier systems it beat.

Sit with the implication: on a task that genuinely mattered, specialization beat scale. The moat wasn't a bigger model — it was the labeled judgment of experts who do this work. That reframes where the value sits in an AI workflow:

• The scarce asset is high-quality expert-labeled data, not access to the biggest model.
• For narrow, repeated, high-stakes tasks, a small tuned model can outperform a frontier one and cost a fraction to operate.
• "Judgment as an asset" is a real competitive edge: firms sitting on decades of expert decisions can encode that into a model nobody else can replicate, precisely because it was never public.

What this means if you're investing in the "AI replaces knowledge work" thesis

Plenty of market capital is priced on the assumption that off-the-shelf frontier models will swallow professional white-collar tasks wholesale. This test is a useful reality check — not a rebuttal, a refinement:

1. The last 20% is the expensive part. Getting from 50% to 78% took expert prompting. Getting from 78% to a trustworthy 85% took proprietary data and fine-tuning. In real deployments, that final gap to reliability is where most of the cost and time actually live.
2. Value accrues to data owners, not just model makers. If specialization wins on the tasks that matter, incumbents with proprietary expert data may capture more of the upside than a pure "best model wins" story suggests.
3. Benchmarks are marketing, deployments are truth. Treat public scores as a floor for capability, never a promise of on-the-job performance.

None of this says AI can't do the work. It says the version that can be trusted looks less like "plug in GPT" and more like "tune a specialist on judgment you already own."

Frequently Asked Questions (FAQ)

Did GPT and Claude "fail" in an absolute sense? No. They performed reasonably (around 78% with expert prompting) — they just fell short of the specific 80% bar Bridgewater set for trusting automation in its own workflow. It's a threshold miss, not a collapse.

Why did naive prompts score only ~50%? Because the task depended on Bridgewater's private notion of relevance. Without careful, expert-written instructions, the models had no way to infer what "relevant" meant in that context, so they landed near chance.

Does this prove fine-tuning always beats frontier models? No — it shows that for a narrow, high-stakes task backed by proprietary expert data, a tuned specialist can beat a general model. For open-ended or novel tasks, frontier models still lead.

Is this an independent public benchmark? No. These are company-run measurements on Bridgewater's own tasks, published jointly with Thinking Machines Lab. That's a strength for realism but means the numbers reflect one firm's specific workflow.

What's the practical lesson for a company adopting AI? Don't buy on leaderboard scores. Test on your real tasks, and budget for the expert-labeling and tuning it takes to reach a reliability bar you'd actually trust.

Key Takeaways

• On real document-filtering tasks, the best frontier model reached 78.2% — under the 80% trust threshold; naive prompts scored ~50%.
• A custom fine-tuned model trained on Bridgewater's private expert labels hit 84.7%, beating the frontier model.
• The models struggled because the correct answers were never public — exposing how much benchmark performance leans on memorized public data.
• The scarce, valuable asset is expert-labeled judgment, not access to the largest model — specialization beat scale here.
• For anyone betting on AI replacing knowledge work: the reliable version usually requires proprietary data and tuning, and that last gap to trust is where the cost is.

참고자료 - CNBC (Oct 2024 context on model testing) / The Decoder: "GPT and Claude failed Bridgewater's finance tests because the right answers were never public" — https://the-decoder.com - WinBuzzer: "Fine-Tuned Alibaba Qwen AI Model Outperforms Claude, GPT, Gemini in Finance Tasks" — https://winbuzzer.com/2026/07/04/bridgewater-test-says-gpt-claude-lag-tuned-qwen-model-xcxwbn/ - FourWeekMBA: "Thinking Machines Lab and Bridgewater Prove a Small Custom Model Beats GPT, Claude, and Gemini on Finance Tasks" — https://fourweekmba.com/ai-thinking-machines-lab-bridgewater-custom-model-beats-frontie/ - Victorino Group: "Judgment as an Asset: What Bridgewater's Fine-Tuned Finance Model Proved" — https://victorinollc.com/thinking/differentiated-intelligence-expert-judgment-finance