Falconer outscores leading AI tools on real support and engineering questions
Benchmarks for docs and code
We judged performance against 200 real questions from a public support corpus and the codebase of a popular open-source project. Falconer wins head-to-head against all leading tools.
Wix help center
Can a system answer real customer support questions using only a help center? We pointed each tool at the same 6,221 Wix help articles, then scored it on 100 real customer questions from the public WixQA set.
Apache Spark
Can a system answer real engineering questions from live source code? We indexed the apache/spark repository and its markdown docs, then scored each tool on 100 Spark questions from Stack Overflow.
Benchmark results
Note: Falconer, Claude Code, and Codex ran at the same thinking-effort level on each test: medium effort on the docs test, high effort on the code test. Notion and Atlassian Rovo do not expose a comparable effort setting.
On real support questions, Falconer wins against all four tools.
Head-to-head win rate
Falconer wins the majority of decisive verdicts in every matchup.
Win share of decisive judge verdicts (6 per question), scored by the weighted-sum rule. Ties excluded; the full record is shown beside each matchup.
Three scoring methods
Matchups were scored holistically, by weighted formula, and by strict Pareto rule.
We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.
| Scoring rule | How it decides | Definition of tie | vs | vs | vs | vs |
|---|---|---|---|---|---|---|
| Weighted-sumused above | Blend the four axes (0.35 faithfulness + 0.35 helpfulness + 0.20 completeness + 0.10 relevance); a side wins if it leads by more than 0.25 on the 0 to 10 scale. | The two scores land within 0.25 of each other. | 70.5% | 88.4% | 52.6% | 62.8% |
| Holistic | Trust the judge's own overall pick of the better answer. | The judge called it a tie. | 70.5% | 87.8% | 55% | 60.2% |
| Pareto | A side wins only if it scores ≥ on all four axes and > on at least one. The strictest rule. | Equal on every quality metric, or mixed (each side better somewhere). | 74% | 94.6% | 51.4% | 52.5% |
Speed
Falconer delivers the fastest full answers: 18.5s median, well ahead of every rival.
Time to full answer and to first token, across the full distribution. Lower and tighter is better.
See exact percentiles
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 19.6s | 15.3s | 18.5s | 23.3s | 30.5s |
| Notion | 27.5s | 22.3s | 27.1s | 31.2s | 35.8s |
| Atlassian Rovo | 31.2s | 26.5s | 30.4s | 34.6s | 37s |
| Claude Code | 29.4s | 21.1s | 27.3s | 36.2s | 41.9s |
| Codex | 27.3s | 21.1s | 25.9s | 31s | 41.3s |
See exact percentiles
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 3.3s | 2.5s | 2.9s | 3.8s | 4.5s |
| Notion | 19.5s | 14s | 18.3s | 23.3s | 30s |
| Atlassian Rovo | 18.4s | 13.8s | 18.9s | 21.4s | 23.7s |
| Claude Code | 2.2s | 1.7s | 2s | 2.3s | 2.9s |
| Codex | 27s | 19.9s | 25.7s | 30.8s | 41.1s |
Answer length
Top performers show shorter answer length.
The judges never reward length; it's shown here only as context.
See exact percentiles
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 1,768 | 1,242 | 1,741 | 2,138 | 2,705 |
| Notion | 2,419 | 1,831 | 2,367 | 2,832 | 3,944 |
| Atlassian Rovo | 5,730 | 5,357 | 6,032 | 6,829 | 7,529 |
| Claude Code | 1,875 | 1,368 | 1,838 | 2,409 | 2,884 |
| Codex | 1,304 | 918 | 1,244 | 1,592 | 2,091 |
On real engineering questions, Falconer wins against all four tools.
Head-to-head win rate
Falconer wins the majority of decisive verdicts in every matchup.
Win share of decisive judge verdicts (6 per question), scored by the weighted-sum rule. Ties excluded; the full record is shown beside each matchup.
Three scoring methods
Matchups were scored holistically, by weighted formula, and by strict Pareto rule.
We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.
| Scoring rule | How it decides | Definition of tie | vs | vs | vs | vs |
|---|---|---|---|---|---|---|
| Weighted-sumused above | Blend the four axes (0.35 faithfulness + 0.35 helpfulness + 0.20 completeness + 0.10 relevance); a side wins if it leads by more than 0.25 on the 0 to 10 scale. | The two scores land within 0.25 of each other. | 57.7% | 97.1% | 56.1% | 74.2% |
| Holistic | Trust the judge's own overall pick of the better answer. | The judge called it a tie. | 58.8% | 96.2% | 55% | 74.2% |
| Pareto | A side wins only if it scores ≥ on all four axes and > on at least one. The strictest rule. | Equal on every quality metric, or mixed (each side better somewhere). | 61.1% | 97.7% | 55.7% | 75.7% |
Speed
On the code test, speed is a near tie: every tool but Codex (72s) returns a full answer in 39 to 45s median.
Time to full answer and to first token, across the full distribution. Lower and tighter is better.
See exact percentiles
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 51.3s | 31.9s | 45s | 64.7s | 85.4s |
| Notion | 45.4s | 28.6s | 40s | 56.2s | 69.8s |
| Atlassian Rovo | 49.3s | 33.2s | 42.6s | 58.3s | 78.3s |
| Claude Code | 45.4s | 25.3s | 39.1s | 55.9s | 86.1s |
| Codex | 78.6s | 58.7s | 72s | 99.8s | 120.8s |
See exact percentiles
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 19.1s | 5.5s | 15.5s | 26s | 42.1s |
| Notion | 27.4s | 12.4s | 24.4s | 36.9s | 50.9s |
| Atlassian Rovo | 13.7s | 11.8s | 12.7s | 15.1s | 17.9s |
| Claude Code | 2.1s | 1.6s | 1.9s | 2.2s | 2.8s |
| Codex | 55.1s | 10.2s | 55.7s | 98.5s | 120.7s |
Answer length
Top performers show shorter answer length.
The judges never reward length; it's shown here only as context.
See exact percentiles
| System | Mean | p25 | Median | p75 | p90 |
|---|---|---|---|---|---|
| Falconer | 3,295 | 2,592 | 3,205 | 4,031 | 4,764 |
| Notion | 3,846 | 2,942 | 3,655 | 4,734 | 5,542 |
| Atlassian Rovo | 586 | 136 | 228 | 452 | 1,676 |
| Claude Code | 3,067 | 2,451 | 3,036 | 3,733 | 4,411 |
| Codex | 1,870 | 1,398 | 1,802 | 2,283 | 2,895 |
How the judges decided
The full dataset is public, including every question, reference answer, assistant response, and judge verdict: FalconerAI/falconer-benchmarks.
Real questions used
The questions come from public, third-party sources: real help-center Q&A from WixQA and Apache Spark questions from Stack Overflow. For Spark, we selected questions by user votes and by whether answering them well requires reading the repository. We published every question, the reference answer, and each assistant’s full response.
Judging principles
Three frontier judges
Every head-to-head is scored by Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro, in both A/B and B/A orderings to neutralize position bias, for six verdicts per question.
A reproducible formula
The headline winner is derived from four metric scores with a fixed weighting. Anyone with the per-metric scores gets the same answer, and the lead holds under two other rules too.
What counts as a tie
Weighted-sum scores within 0.25 of each other count as ties and are excluded from head-to-head win rates.
Pairwise, against a human reference
Every answer is judged head-to-head against Falconer, with the human-written gold answer (the WixQA reference, or the accepted Stack Overflow answer) as the reference both sides are measured against, rather than grading either answer in isolation.
No web access for any agent
Every system runs sealed to its corpus. Web tools are disabled and verified at runtime, so results reflect retrieval and grounding.
Public, reproducible corpora
Both question sets are public (the WixQA support corpus and the apache/spark codebase with its Stack Overflow Q&A), at 100 questions each, so anyone can rerun the same benchmark against their own system.
- The displayed records use the headline weighted-sum rule: a verdict is a tie when the two weighted scores land within 0.25 of each other on the 0 to 10 scale, too close for the judge to separate (the scoring table shows how each of the three rules defines a tie). Head-to-head % counts wins among the decisive (non-tie) verdicts, the standard way win rates are reported.
- Each question set is 100 items drawn from a public benchmark; larger samples tighten the confidence intervals.
- Coverage varies slightly. Notion answered 94 of 100 support questions; on the other 6 it returned an interactive clarification form instead of an answer, so those are excluded. Rovo answered 98 of 100 engineering questions. Win/loss/tie counts are tallied over the questions each system actually answered (6 verdicts per answered question), so they total below 600 for those systems. One Codex verdict on the engineering test was an unparseable judge response and is also excluded, so that record totals 599.
- On the engineering test, Atlassian Rovo is a structural baseline rather than a like-for-like rival: the apache/spark repository was hosted in Bitbucket, which Rovo barely reads, so its short answers reflect missing source access rather than answer quality. Read that matchup as retrieval-with-code vs retrieval-without-code; Notion, Claude Code, and Codex are the code-capable cohort.
- Per-metric averages and head-to-head wins can diverge: a system can post a higher average on one metric yet win fewer questions, because each question’s winner is decided from all four metrics together.
- Ties (weighted-sum scores within 0.25 of each other) are excluded from head-to-head %. Tie rates ranged from ~2% (Rovo) to ~36% (Claude Code); closer matchups naturally produce more ties.
- The Stack Overflow reference is one valid solution, not the only correct answer; judges credit alternate-but-correct approaches.
- Falconer, Claude Code, and Codex run named models at the same thinking-effort level on each test: medium effort on the support test, high effort on the engineering test. Notion exposes its model (Opus 4.8) but not its effort level; Rovo ran in its "Think deeper" mode on an undisclosed model.
- Results are a point-in-time snapshot from a June 2026 run.
Four metrics, deliberately weighted
Every answer scored 0 to 10 against the human reference.
Every claim is correct and supported by the source, with no fabricated steps, wrong API names, invented config keys, or made-up numbers, even when the rest of the answer reads well.
Is every claim true to the source?
Concrete and actionable: real UI paths, working code, specific steps and version numbers. Vague guidance and "check the docs" deflections are penalized.
Can the reader act on it directly?
Covers every critical step or fact needed to actually resolve the question. Tangential background does not raise this score.
Did it cover everything needed to finish the job?
The answer is relevant. Sometimes an answer can be factually true, but irrelevant to the problem in question. Tangents, unrequested background, and citation dumps are penalized. Length itself is not.
How much is on-target signal vs. noise?
score = 0.35·faithfulness + 0.35·helpfulness + 0.20·completeness + 0.10·relevance- Correct and actionable matter most, so faithfulness and helpfulness get 35% each. A fluent answer with the wrong API name is worthless.
- Completeness (20%) catches answers that skip a critical step.
- Relevance (10%) penalizes off-topic padding.
- Completeness and relevance deliberately pull against each other: adding content lifts completeness but costs relevance, so neither rewards bloat.
The real benchmark is your own data
These results come from publicly available data. Falconer is even better when connected to your own code, docs, and sources like Linear, Slack, and meeting notes.