26 specialized decision models and 3 high-end LLM references. Same 1342 text questions, 31 categories. Updated October 21, 2026.
Leading decision-model score
84.9%
Mammoth-1 · 1139/1342
Highest LLM reference score
85.2%
GPT-6 Sol, reasoning off · 1143/1342
Complete matched coverage
29 routes
38,918 first attempts · 31 categories
Tied decision-model scores
84.0%
Span-02 and Drex Max · 1127 each
Accuracy, speed and cost: all 29 routes
Blue bars: decision models. Pale yellow: separately labelled, no-reasoning LLM references. Every row covers 1342 first attempts.
Model
Type
Correct / questions
Median
p95
Cost per 1000
Actual 1342-question cost
GPT-6 Sol — no reasoning
LLM reference
1143/1342 85.2%
1023 ms
1546 ms
$0.632112
$0.848294
Mammoth-1
Decision model
1139/1342 84.9%
240 ms
666 ms
Promotional free access*
Promotional free access*
Drex Max
Decision model
1127/1342 84.0%
286 ms
445 ms
$0.021180
$0.028423
Respan Span-02
Decision model
1127/1342 84.0%
206 ms
2157 ms
$0.003537
$0.004746
Perplexity Decider v1.2
Decision model
1120/1342 83.5%
249 ms
332 ms
$0.007000 + ?
$0.009394 + ?
Nimble v4 Max
Decision model
1115/1342 83.1%
235 ms
329 ms
$0.027633
~$0.037084
d3-turbo
Decision model
1106/1342 82.4%
165 ms
298 ms
Free public access*
Free public access*
Liquid d1-ultra
Decision model
1084/1342 80.8%
200 ms
393 ms
$0.012124 + ?
$0.016270 + ?
Amos 1 Large
Decision model
1074/1342 80.0%
311 ms
360 ms
Free public access*
Free public access*
Sonnet 5.5 — no thinking, JSON-choice
LLM reference
1070/1342 79.7%
1506 ms
2424 ms
$1.433613
$1.923908
kettei-fat
Decision model
1066/1342 79.4%
174 ms
214 ms
$0.015433
$0.020711
Jev 1.14
Decision model
1030/1342 76.8%
179 ms
269 ms
$0.024571
$0.032974
Jev 1.13
Decision model
1026/1342 76.5%
189 ms
323 ms
$0.024571
$0.032974
Mercury Decide
Decision model
1017/1342 75.8%
238 ms
388 ms
$0.006956
$0.009334
Elephant-1
Decision model
1004/1342 74.8%
139 ms
272 ms
Promotional free access*
Promotional free access*
Qwen3.8-Max — none, Hyper Charm
LLM reference
967/1342 72.1%
1191 ms
1731 ms
$0.614821 + ?
$0.825090 + ? quota
Liquid d1
Decision model
965/1342 71.9%
206 ms
827 ms
$0.005773
$0.007748
Cloudflare Clef
Decision model
936/1342 69.7%
268 ms
417 ms
$0.017594
$0.023611
Perplexity Decider Flash v1.2
Decision model
923/1342 68.8%
159 ms
231 ms
$0.003406 + ?
$0.004570 + ?
Microsoft-Decision-1
Decision model
921/1342 68.6%
283 ms
566 ms
$0.012237
$0.016422
Drex v2.0
Decision model
915/1342 68.2%
232 ms
755 ms
$0.016661 + ?
$0.022359 + ?
Cloudflare Clef Flash
Decision model
860/1342 64.1%
245 ms
353 ms
$0.008797
$0.011806
Cloudflare Clef Omni
Decision model
860/1342 64.1%
241 ms
400 ms
$0.061511 + ?
$0.082547 + ?
Drex v1.5
Decision model
853/1342 63.6%
203 ms
649 ms
$0.011160
$0.014976
Kev 4B
Decision model
822/1342 61.3%
225 ms
377 ms
$0.011718
$0.015725
Luna 6 Decisions
Decision model
758/1342 56.5%
254 ms
527 ms
$0.035840 + ?
$0.048097 + ?
Tev1 4B experimental (OpenRouter native adapter)
Decision model
751/1342 56.0%
198 ms
253 ms
$0.015657
$0.021011
ScaleDown Classify-1
Decision model
550/1342 41.0%
268 ms
362 ms
$0.021388
$0.028703
Solar Decide Flash
Decision model
538/1342 40.1%
540 ms
747 ms
$0.030203 + ?
$0.040532 + ?
*Free public or promotional access is not an API-reported $0 receipt or a permanent production price. No numeric free-price point is plotted. Baseten’s launch promotion is temporary. Costs per 1000 are normalized from actual 1342-question totals; + ? preserves unreported charges. Qwen is a prepaid-quota valuation.
Fixed speed and cost color bands
Band
Response time
Known cost per 1,000
Green
≤250 ms
≤$0.010
Yellow
>250–500 ms
>$0.010–0.025
Orange
>500–1,000 ms
>$0.025–0.050
Red
>1,000 ms
>$0.050
Thresholds use unrounded values. Incomplete costs describe a known lower bound. The median, not p95, is the speed-screening measure.
October 21: Mammoth-1, Elephant-1 and Drex Max
Model
Correct /1342
Average (mean)
Median
p95
Actual run cost / access
Route
Mammoth-1
1139/1342 ·84.9%
301.4 ms
239.6 ms
665.6 ms
Promotional free access*
Direct → Baseten → decision-models/baseten-mammoth-27b-latest
Elephant-1
1004/1342 ·74.8%
163.9 ms
139.2 ms
271.9 ms
Promotional free access*
Direct → Baseten → decision-models/baseten-elephant-9b-latest
Drex Max
1127/1342 ·84.0%
289.8 ms
285.8 ms
444.8 ms
$0.028423
OpenRouter → DeepInfra → nace-ai/drex-max
All 4026 new first attempts were usable, without recoveries or probability-format warnings. Neither partial OpenRouter control is ranked. Earlier 26 model records are unchanged.
Mammoth-1 leads the decision-model sample
Mammoth-1 answered 1139/1342 correctly (84.9%), twelve more than Span-02 and Drex Max, which tied at 1127/1342 (84.0%). GPT-6 Sol without reasoning answered 1143 correctly (85.2%), four more than Mammoth. The four-answer gap does not establish a reliable general ordering. Mammoth’s median response was 240 ms and Sol’s was 1023 ms: about 4.3 times longer for Sol on these measured routes. Sonnet’s separately constrained, no-thinking reference remains 1070/1342 (79.7%).
The leader still has specific weak operations
Mammoth scored 64/67 on one-to-one assignments—matching each person or object to exactly one available slot—and 57/58 on exceptions and priority rules. Tracking what others know gave 47/50. These tasks can ask where someone will search after an object moved while they were absent. Counting exactly true claims gave 44/71, truth-tellers and liars 48/74, and ordered procedures only 18/48. Procedures require applying each edit to the current state, not merely recognizing a plausible final answer. Span scored 33/48 there, despite its lower overall total. Mammoth also scored 38/52 on tracking swaps, versus Span’s 28/52.
Drex Max ties Span overall, with different strengths
Drex Max scored 65/67 on one-to-one assignments and 45/46 on if-then rules, versus Span’s 63/67 and 32/46. On procedures, Drex scored 18/48 and Span 33/48; on worst-case choices without supplied probabilities, Drex scored 5/8 and Span 8/8. Equal overall totals do not mean the same decisions will be correct. Drex’s median/p95 was 286/445 ms, versus Span’s previously measured 206/2157 ms: Span had the shorter median, Drex the shorter tail. Drex’s complete API-reported cost was $0.02842302, normalized to $0.02118 per 1000 questions. Span’s recorded $0.00474628 total was about six times lower.
Elephant-1 is fast, but not a uniform improvement over Jev
Elephant-1 scored 1004/1342 (74.8%), below Jev 1.14’s 1030/1342 (76.8%). Its 139 ms median was shorter than Jev’s 179 ms. Elephant still scored 60/67 on one-to-one assignments and 46/50 on knowledge tracking, versus Jev’s 57/67 and 44/50. Comparing time intervals gave Elephant 25/42 and Jev 40/42; checking required conditions gave 38/52 and 48/52. A newer release is not automatically stronger across this bank.
The added questions expose more separation
On the original 1136 questions, Mammoth scored 1019/1136 (89.7%), Drex Max 1004/1136 (88.4%) and Elephant 902/1136 (79.4%). On the 206 additions their scores were 120/206 (58.3%), 123/206 (59.7%) and 102/206 (49.5%). This is a descriptive split, not a second leaderboard or a change in weighting. Across the six new categories, Mammoth scored 30/48, Drex 29/48 and Elephant 22/48. Each new category contains only eight questions; one answer moves its score by 12.5 percentage points.
Jev’s position is a measured result, not an explanation of its roadmap
Mammoth answered 109 more questions correctly than Jev 1.14; Drex Max answered 97 more. Jev still answered all 46 if-then questions correctly, versus 45 each for Mammoth and Drex. This bank measures supplied-fact, five-choice decisions without tools or agentic iteration; it does not test broad knowledge, multilingual coverage or product integration. It cannot establish why a developer has or has not improved a model.
Free launch access is separate from a production price
Mammoth and Elephant used Baseten’s direct API at the captured promotional zero tariff. Their responses did not contain monetary receipts, so the report does not invent an API-reported $0 price or plot one on the logarithmic cost axis. Baseten advertises a one-week launch promotion; availability and later prices may change. Their 20-question OpenRouter controls agreed with the direct route on all 20 choices for each model. Those controls are not pooled into the full-bank scores or latency measurements.
High-end LLM references, reasoning off
These three selected reference routes show the level of immediate decision performance alongside generative models. They are not an exhaustive ranking of every LLM that can disable reasoning. No tools or iterative repair; 16 output tokens maximum.
Sol: OpenRouter → OpenAI, explicit reasoning off. Sonnet: OpenRouter → Anthropic, no tools, constrained JSON-choice schema. Both complete runs report zero reasoning/thinking tokens. Sonnet’s separate bare-ID trial produced 84 unusable outputs among 110 and is not pooled into its complete score. Qwen: Hyper Charm, reasoning_effort=none; the route does not report reasoning-token counts, so missing telemetry is not called zero. Its original delivery failures remain in the denominator.
Category scores: what each task means
All 31 categories are arranged into five reading groups. These groups organize the display only; they do not change the benchmark, its weighting or any score.
How to read these results
Read a row to compare one model across the named tasks. Read a column to compare models on one task.
Each cell gives correct answers / questions, followed by the percentage. For example, 6/8 · 75% means six correct first decisions out of eight, not a confidence rating.
Darker blue means higher accuracy on the same fixed 0-100% scale throughout. Random guessing among five choices averages 20%. Percentages help compare differently sized categories; counts show the size of the evidence.
The model names stay visible when you scroll sideways. On a narrow screen, each explanation card also has an All 29 routes list with no wide matrix to decode.
Small categories are coarse: one answer changes an eight-question category by 12.5 percentage points. A perfect sampled score is not a reliability guarantee.
0%20%40%60%80%100%
Rules and evidence
Which facts meet a rule, which conclusions follow, and when is the evidence insufficient?
These decisions involve uncertain outcomes, another actor, overlapping operations or possible information leaks.
“Without probabilities” means exactly that: the question gives possible situations and their costs, but does not say how likely each situation is. The model must use the specified worst-case loss or regret rule, rather than inventing an average.
Decide whether paying for a test before choosing an action reduces the probability-weighted cost enough to cover the test price.
Example: Should you buy a sensor reading before choosing an action, or is acting without the extra information cheaper overall?
Highest observed: GPT-6 Sol (LLM, none) · 6/8 (75.0%)
All 29 routes in this category
Model
Correct / questions (%)
GPT-6 Sol — no reasoning
6/8 (75.0%)
Respan Span-02
5/8 (62.5%)
Liquid d1-ultra
5/8 (62.5%)
Drex v1.5
5/8 (62.5%)
Mammoth-1
4/8 (50.0%)
Drex Max
4/8 (50.0%)
Perplexity Decider v1.2
4/8 (50.0%)
d3-turbo
4/8 (50.0%)
Sonnet 5.5 — no thinking, JSON-choice
4/8 (50.0%)
kettei-fat
4/8 (50.0%)
Jev 1.14
4/8 (50.0%)
Elephant-1
4/8 (50.0%)
Qwen3.8-Max — none, Hyper Charm
4/8 (50.0%)
Perplexity Decider Flash v1.2
4/8 (50.0%)
Microsoft-Decision-1
4/8 (50.0%)
Drex v2.0
4/8 (50.0%)
Cloudflare Clef Flash
4/8 (50.0%)
Luna 6 Decisions
4/8 (50.0%)
Nimble v4 Max
3/8 (37.5%)
Amos 1 Large
3/8 (37.5%)
Jev 1.13
3/8 (37.5%)
Mercury Decide
3/8 (37.5%)
Liquid d1
3/8 (37.5%)
Cloudflare Clef
3/8 (37.5%)
Cloudflare Clef Omni
3/8 (37.5%)
ScaleDown Classify-1
2/8 (25.0%)
Solar Decide Flash
2/8 (25.0%)
Kev 4B
1/8 (12.5%)
Tev1 4B experimental (OpenRouter native adapter)
0/8 (0.0%)
Worst-case choices without probabilities
F28 · 8 questions
Follow the stated worst-case rule when likelihoods are not supplied. Minimize either the largest possible loss or the largest regret (extra loss compared with the best action for each situation), as the question specifies. Do not invent probabilities.
Example: A route costs 4 in either situation; another costs 1 or 9. With no probabilities and a smallest-worst-case-cost rule, which route should be chosen?
Highest observed: Respan Span-02 · 8/8 (100.0%)
All 29 routes in this category
Model
Correct / questions (%)
Respan Span-02
8/8 (100.0%)
GPT-6 Sol — no reasoning
6/8 (75.0%)
Drex Max
5/8 (62.5%)
Cloudflare Clef Flash
5/8 (62.5%)
Mammoth-1
4/8 (50.0%)
Nimble v4 Max
4/8 (50.0%)
Liquid d1-ultra
4/8 (50.0%)
Sonnet 5.5 — no thinking, JSON-choice
4/8 (50.0%)
Qwen3.8-Max — none, Hyper Charm
4/8 (50.0%)
Kev 4B
4/8 (50.0%)
Luna 6 Decisions
4/8 (50.0%)
Perplexity Decider v1.2
3/8 (37.5%)
Amos 1 Large
3/8 (37.5%)
kettei-fat
3/8 (37.5%)
Mercury Decide
3/8 (37.5%)
Elephant-1
3/8 (37.5%)
Cloudflare Clef
3/8 (37.5%)
Drex v2.0
3/8 (37.5%)
Cloudflare Clef Omni
3/8 (37.5%)
d3-turbo
2/8 (25.0%)
Jev 1.14
2/8 (25.0%)
Jev 1.13
2/8 (25.0%)
Liquid d1
2/8 (25.0%)
Perplexity Decider Flash v1.2
2/8 (25.0%)
Microsoft-Decision-1
2/8 (25.0%)
Drex v1.5
2/8 (25.0%)
Tev1 4B experimental (OpenRouter native adapter)
2/8 (25.0%)
ScaleDown Classify-1
1/8 (12.5%)
Solar Decide Flash
0/8 (0.0%)
Anticipating another decision-maker
F29 · 8 questions
Choose an action while accounting for the specified choices or payoff-maximizing response of another decision-maker.
Example: Which first move gives the best result after the other player chooses their own best reply?
Highest observed: Mammoth-1 · 6/8 (75.0%)
All 29 routes in this category
Model
Correct / questions (%)
Mammoth-1
6/8 (75.0%)
Drex Max
5/8 (62.5%)
Respan Span-02
5/8 (62.5%)
Nimble v4 Max
5/8 (62.5%)
d3-turbo
5/8 (62.5%)
kettei-fat
5/8 (62.5%)
Jev 1.14
5/8 (62.5%)
Mercury Decide
5/8 (62.5%)
GPT-6 Sol — no reasoning
4/8 (50.0%)
Perplexity Decider v1.2
4/8 (50.0%)
Liquid d1-ultra
3/8 (37.5%)
Amos 1 Large
3/8 (37.5%)
Jev 1.13
3/8 (37.5%)
Elephant-1
3/8 (37.5%)
Qwen3.8-Max — none, Hyper Charm
3/8 (37.5%)
Liquid d1
3/8 (37.5%)
Cloudflare Clef
3/8 (37.5%)
Perplexity Decider Flash v1.2
3/8 (37.5%)
Microsoft-Decision-1
3/8 (37.5%)
Drex v2.0
3/8 (37.5%)
Cloudflare Clef Flash
3/8 (37.5%)
Cloudflare Clef Omni
3/8 (37.5%)
Kev 4B
3/8 (37.5%)
Tev1 4B experimental (OpenRouter native adapter)
3/8 (37.5%)
Drex v1.5
2/8 (25.0%)
Luna 6 Decisions
2/8 (25.0%)
Solar Decide Flash
2/8 (25.0%)
Sonnet 5.5 — no thinking, JSON-choice
1/8 (12.5%)
ScaleDown Classify-1
1/8 (12.5%)
Checking overlapping operations
F30 · 8 questions
Decide whether overlapping operations and their reported results can fit a permitted one-at-a-time order that respects the specified timing and object rules.
Example: Two processes read and change the same stored value. Could their reported results fit an allowed one-at-a-time order?
Highest observed: GPT-6 Sol (LLM, none) · 7/8 (87.5%)
All 29 routes in this category
Model
Correct / questions (%)
GPT-6 Sol — no reasoning
7/8 (87.5%)
Mammoth-1
5/8 (62.5%)
Respan Span-02
5/8 (62.5%)
Sonnet 5.5 — no thinking, JSON-choice
5/8 (62.5%)
kettei-fat
5/8 (62.5%)
Mercury Decide
5/8 (62.5%)
Cloudflare Clef
5/8 (62.5%)
Perplexity Decider v1.2
4/8 (50.0%)
d3-turbo
4/8 (50.0%)
Jev 1.14
4/8 (50.0%)
Jev 1.13
4/8 (50.0%)
Elephant-1
4/8 (50.0%)
Qwen3.8-Max — none, Hyper Charm
4/8 (50.0%)
Microsoft-Decision-1
4/8 (50.0%)
Drex v2.0
4/8 (50.0%)
Drex v1.5
4/8 (50.0%)
Solar Decide Flash
4/8 (50.0%)
Drex Max
3/8 (37.5%)
Nimble v4 Max
3/8 (37.5%)
Liquid d1-ultra
3/8 (37.5%)
Amos 1 Large
3/8 (37.5%)
Liquid d1
3/8 (37.5%)
Perplexity Decider Flash v1.2
3/8 (37.5%)
Kev 4B
3/8 (37.5%)
Luna 6 Decisions
3/8 (37.5%)
Tev1 4B experimental (OpenRouter native adapter)
3/8 (37.5%)
Cloudflare Clef Flash
2/8 (25.0%)
Cloudflare Clef Omni
2/8 (25.0%)
ScaleDown Classify-1
1/8 (12.5%)
Detecting secret-information leaks
F31 · 8 questions
Check whether changing a protected secret can change an observable result in a way the stated policy forbids.
Example: If public inputs stay the same, could public output distinguish two secret values that the policy says must remain indistinguishable?
Highest observed: GPT-6 Sol (LLM, none) · 8/8 (100.0%)
All 29 routes in this category
Model
Correct / questions (%)
GPT-6 Sol — no reasoning
8/8 (100.0%)
Drex Max
7/8 (87.5%)
Sonnet 5.5 — no thinking, JSON-choice
7/8 (87.5%)
Qwen3.8-Max — none, Hyper Charm
7/8 (87.5%)
Mammoth-1
6/8 (75.0%)
Perplexity Decider v1.2
6/8 (75.0%)
Nimble v4 Max
6/8 (75.0%)
d3-turbo
6/8 (75.0%)
Amos 1 Large
6/8 (75.0%)
kettei-fat
6/8 (75.0%)
Jev 1.14
6/8 (75.0%)
Cloudflare Clef
6/8 (75.0%)
Microsoft-Decision-1
6/8 (75.0%)
Respan Span-02
5/8 (62.5%)
Liquid d1-ultra
5/8 (62.5%)
Mercury Decide
5/8 (62.5%)
Elephant-1
5/8 (62.5%)
Liquid d1
5/8 (62.5%)
Jev 1.13
4/8 (50.0%)
Perplexity Decider Flash v1.2
4/8 (50.0%)
Drex v2.0
4/8 (50.0%)
Cloudflare Clef Flash
4/8 (50.0%)
Cloudflare Clef Omni
4/8 (50.0%)
Drex v1.5
4/8 (50.0%)
Kev 4B
4/8 (50.0%)
Luna 6 Decisions
4/8 (50.0%)
Tev1 4B experimental (OpenRouter native adapter)
4/8 (50.0%)
Solar Decide Flash
4/8 (50.0%)
ScaleDown Classify-1
3/8 (37.5%)
View or download the full-size chart for this group
Download the full named-category score matrix (CSV). Display names use plain language; stable category IDs and the tested data are unchanged. Examples explain the task types; they are illustrations, not additional scored questions.
Delivered decisions and failures
Model
Usable first
Wrong valid
Provider failures
Output failures
Recoveries
Eventual correct
GPT-6 Sol — no reasoning
1342
199
0
0
0
1143
Mammoth-1
1342
203
0
0
0
1139
Drex Max
1342
215
0
0
0
1127
Respan Span-02
1342
215
0
0
0
1127
Perplexity Decider v1.2
1341
221
1
0
1
1121
Nimble v4 Max
1342
227
0
0
0
1115
d3-turbo
1342
236
0
0
0
1106
Liquid d1-ultra
1341
257
1
0
1
1085
Amos 1 Large
1342
268
0
0
0
1074
Sonnet 5.5 — no thinking, JSON-choice
1342
272
0
0
0
1070
kettei-fat
1342
276
0
0
0
1066
Jev 1.14
1342
312
0
0
0
1030
Jev 1.13
1342
316
0
0
0
1026
Mercury Decide
1342
325
0
0
0
1017
Elephant-1
1342
338
0
0
0
1004
Qwen3.8-Max — none, Hyper Charm
1317
350
25
0
23
971
Liquid d1
1342
377
0
0
0
965
Cloudflare Clef
1342
406
0
0
0
936
Perplexity Decider Flash v1.2
1286
363
56
0
56
940
Microsoft-Decision-1
1342
421
0
0
0
921
Drex v2.0
1302
387
40
0
40
933
Cloudflare Clef Flash
1342
482
0
0
0
860
Cloudflare Clef Omni
1341
481
1
0
1
861
Drex v1.5
1342
489
0
0
0
853
Kev 4B
1342
520
0
0
0
822
Luna 6 Decisions
1300
542
42
0
42
761
Tev1 4B experimental (OpenRouter native adapter)
1342
591
0
0
0
751
ScaleDown Classify-1
1342
792
0
0
0
550
Solar Decide Flash
1340
802
2
0
2
539
Image input is a separate test
Mammoth-1 and Elephant-1 are text-only on the tested catalog routes. Drex Max documents image input through its direct Nace API; no image score is inferred here. The separate image benchmark and its measured results are unchanged.
Method, cost and measurement limits
The comparison contains 26 specialized decision models and three no-reasoning LLM reference routes: 29 × 1342 = 38,918 first attempts. All earlier 26 model records are preserved exactly. New full-bank runs: October 21, 2026, America/Toronto; Mammoth and Elephant direct through Baseten, Drex Max through OpenRouter’s pinned DeepInfra endpoint.
Same frozen 1342-question bank, seed 2026101906, original five choices and native inputs. Every first attempt remains in the accuracy and latency denominator, including failures. One recovery is allowed only for transport failures or HTTP 408/429/5xx; no answer-dependent retries. All 4026 new first attempts returned usable decisions, with zero recoveries and zero probability-format warnings. After at least 20 questions, a cumulative median above 2000 ms stops a route; none of the three reached that threshold.
Response timing starts before a fresh HTTPS connection and ends after the full response body. Preparation, deliberate pacing, queue waits and recovery delays are outside the response timer. Two requests per model run in barrier waves, with up to three models active. Provider hardware, dates, loads and routes are not controlled away; latency is not batch throughput or an isolated inference-speed measurement.
Baseten’s catalog and returned latest serving aliases identify Mammoth-1 and Elephant-1, not immutable weights. Drex returned nace-ai/drex-max-20261016 from DeepInfra. Baseten’s official technical report describes one-pass option-logit decisions. These native requests ask for neither reasoning nor explanations. Reported native output-token counters are retained; a nonzero counter alone is not evidence of thinking. Undisclosed internal execution cannot be independently verified from an API envelope.
Drex uses reported usage.cost for every attempt. Baseten promotional access is labelled separately from metered receipts. Separate compatibility probes and the two 20-case OpenRouter controls are excluded from the 4026 ranked first attempts. Missing charges remain unknown, not zero. Existing Nimble price estimates and Qwen prepaid-quota valuations retain their original labels.
This is a developing benchmark with uneven category sizes and dependent variants, not 1342 independent observations or a pristine held-out evaluation. The 37 permanent author-development exclusions retain their separate 1305-case diagnostic. Two audited ambiguous original items were already replaced before these runs. All 206 additions retain their existing independent verification and computational evidence; the original bank received a targeted difficult-item audit, not independent external verification of every original question.
Public exports contain unchanged answer-free question inputs, aggregate category scores and sanitized evidence hashes—not answer keys, per-item correctness, private verifier outputs, raw account state or credentials. Public checks verify consistency but cannot reproduce private-key scoring or raw-latency pooling. REAX 1 is not included: a hosted API for that exact fine-tune has not been confirmed.
