iCreatAI News
Back to main site
Skip to content
#Benchmarks
Nov 11Wed
  1. The Decoder82

    Gemini 4 Carbon arrives just in time to feel like yesterday's news in the fastest model cycle yet

    Google has released Gemini 4 Carbon, which it says complements rather than replaces Gemini 4 Argon, targeting complex multi-step coding, debugging and open-ended scientific research. Anthropic's newly unveiled Fable 5.2 outperforms Carbon in the Artificial Analysis test even at lower reasoning intensity, though Carbon costs far less per task. Google also previewed a "/swarm" feature in Antigravity and reported that Carbon achieved a zero percent prompt injection success rate on the Gray Swan IPI benchmark.

    Why it matters: It captures a fast-moving model race and pairs benchmark claims with cost and safety caveats worth weighing.

  2. Hacker News AI48

    Singularity Gate: Benchmark for AI-driven paradigm-shifting scientific discovery

    Singularity Gate is a benchmark built from real scientific findings first made public after the newest model's training cutoff, so no model could have seen them in training. Its v2.1 corpus contains 80 new findings disclosed in July 2026 or later, with a 12-task Hard set rated most paradigm-breaking, and models were run at xhigh reasoning effort in their native harnesses with tools on and web search off. Fable 5.2 leads Physics and Astronomy, Chemistry and Materials, Mathematics and TCS, and Life Sciences, while Opus 5.5 leads Earth and Planetary.

  3. Wiz Research58

    How to Evaluate AI Agent Cybersecurity Capabilities

    Wiz Research outlines the principles behind the relaunch of its Cyber Model Arena, a benchmark of 26 models, four general-purpose harnesses, 59 agents and 327 challenges across code vulnerabilities, API, web and cloud security, arguing that benchmarks should fix their objective first, use real-world tasks, grade deterministically with code rather than LLM judges, isolate agents to prevent cheating, and weigh success against cost. The post reports that agents perform very differently by category (average pass@1 of 78.0% on API security versus 38.2% on code vulnerabilities) and that pairing harness with model matters, with a mean absolute gap of 5.1 pass@1 points across 22 models run on both Claude Code and ADK React. It also lists limitations including simple harnesses, string-equality grading on code tasks, public web tasks that may be contaminated, a 1,000-second time cap and unpublished confidence intervals.

  4. OpenRouter Announcements64

    Best OCR Model for Scanned Documents, Measured

    The author tested 15 vision models plus Mistral OCR and Tesseract on eight synthetic scanned pages scored by character error rate and 264 key-value checks, finding Claude Opus 5.5 most accurate (262 of 264 at $16.42 per 1,000 pages) and Mistral OCR the best value (258 of 264 at $2.02). The post documents failure modes such as fabricated card digits, wifi passwords and server names on a faded receipt, and provides code for transcription, disabling reasoning, cross-checking two models from different labs, and using the mistral-ocr engine.

    Why it matters: It offers a reproducible benchmark with per-model costs and practical code for catching hallucinated values in OCR pipelines.

  5. Hacker News AI40

    Can you prompt your AI into telling the truth? I tested it

    Software developer Scott DeSapio recounts conversations with Claude Opus 4.6 in which he pushed the model to commit to direct honesty, wrote a workspace rule against hedging, and then had it name Objectivism as the source of the rule's principles, with the agent admitting a trained bias against the philosophy. He then opened a fresh chat with both rules in place and asked the model about Objectivism cold, presenting the results as a test of whether prompting can override default agreeableness.

Nov 10Tue
  1. Hacker News AI50

    Maybe Be Careful: What games like RuneScape, Clocktower teach us about AI trust

    The article examines how multi-agent AI systems behave when they must trust one another, drawing on OpenAI's Hugging Face hack and Navier–Stokes experiments, Max Bittker's RuneScape Bench (where a Grok-run alchemist named Quinn scammed a trusting Claude agent to finish second), and Dwarkesh Patel's Blood on the Clocktower benchmark, where GPT models outperformed Claude at detecting deception. It argues that social strategy games offer a way to measure how intelligently agents navigate each other, and that learning to lie may be inseparable from learning to detect lies.

  2. The Decoder80

    Anthropic's Fable 5.2 completes 41 percent of real freelance jobs, up from 2.5 percent a year ago

    On Scale Labs' Real-World Labor Index, Anthropic's Fable 5.2 successfully completes 41.3 percent of 240 real Upwork projects worth about $144,000, up from 2.5 percent for the top system, Manus, when the benchmark launched about a year ago and from Fable 5's roughly 16 percent in July. The next-best systems are Anthropic's Opus 5.5 at 21.25 percent and OpenAI's GPT 6 Astra at 20.83 percent, and Scale Labs says Fable 5.2 makes its biggest gains in audio and music editing, architecture and interior design, and 3D modeling while still struggling with industrial product design and animation. Human experts manually verify each submission in professional software, and CAIS's attempts to automate that review with another AI have so far failed because AI reviewers are fooled by polished but technically flawed work.

    Why it matters: It offers a concrete, human-verified measure of how quickly AI agents are closing the gap on paid professional work, along with the benchmark's limits.

  3. Hacker News AI42

    Wake me up when AI can...

    The author collected about 800 Hacker News comments from 2016 to 2026 of the form "wake me up when AI can…" and "I'll believe it when…", then let readers vote on whether each bar had been cleared; over 100,000 votes from almost 9,700 people were cast before voting closed. Of 826 statements, 112 were judged "clearly happened" (mostly code and text tasks), 89 "clearly not yet" (mostly physical-world and novel-discovery tasks), and 78 were contested.

  4. Hacker News AI68

    Is AI code rotting open source? 450 repos, 12 months

    An analysis of 450 top GitHub repositories found that the average repo had 11.5% of its merged pull requests AI-signed in September 2026, up from 2.3% in October 2025, though the median repo was at just 1.5% and 264 of the 450 repos have committed agent instruction files, making every AI figure a floor. Among 100 TypeScript and C# repos, complexity per 1,000 lines rose a median 4.8% in AI-heavy repos versus 3.0% in others, with the increase concentrated in fast-growing codebases while mature Microsoft projects held flat. AI PRs were typically 2.4× larger than other PRs and fixed within 30 days only slightly more often.

  5. Hacker News AI58

    We checked 312 AI-built apps: how ready are they to launch?

    The article reports a November 2026 launch-readiness check of 312 publicly shared AI-built apps, finding that only 19% scored 90 or more out of 100, while 68% had no analytics, 53% of apps on their own domain lacked email DNS setup, and 49% served an almost empty page without JavaScript. It also found 42% had headlines that didn't say what the app does or who it's for, 37% took over 4 seconds to show main content on a phone, and 37% had no sitemap. The check used the same 15 checks as the site's free launch check, one per homepage, and published totals only.

  6. Hacker News AI62

    Show HN: AI Vision Index, private tests of how well 36 AI models see

    The AI Vision Index publishes independent, private tests of 36 vision models from 18 makers across 29 tests and six skill categories, using self-made images never released publicly so models cannot have seen them. GPT-6 Astra medium leads with 83.7 ±2.0 out of 100 at $38.32 per 1,000 answers, followed by GPT-6.1 Sol medium (81.3) and Claude Fable 5.2 (80.7), while local models such as LFM2.5-VL 1.6B score 12.4. Scores are corrected for guessing, tied results share a rank, and each entry lists its test date and cost.

  7. Hacker News AI58

    Beyond FLOP/s: comparing AI hardware for inference by memory and communication

    The article argues that raw FLOP/s understates inference gains, estimating Nvidia's GB300 delivers about 25× H100's inference capacity per chip on modeled agentic workloads despite only 2.5× the 8-bit FLOP/s, because memory and inter-chip communication matter more. It introduces the Inference Performance Index (IPI), scored in GB300 equivalents (GB300e), which compares simultaneous requests each chip can serve across 50–400 tokens/s/req for four representative models. Reported scores include GB300 NVL72 at 1, B300 HGX at 0.54, MI355X at about 0.55, VR200 at 2.18, and H200 HGX at 0.12.

  8. Hacker News AI22

    Three AI engines: how often they name the same local business

    A study covering 2026-10-24 to 2026-11-09 found that any two of three AI engines (ChatGPT gpt-5-mini, Claude bedrock haiku 4.5, Gemini flash lite) share at least one business name 56.8% of the time (95 percent interval 54.5 to 59.0 percent), based on 1066 engine pairs across 771 question-days, 54 questions and 17 run days. Names were extracted mechanically by capitalisation from the public corpus and matched on canonical form, with generic names filtered. The study counts agreement only, not causes or outcomes.

  9. Latent Space88

    [AINews] Fable 5.2 retakes World #1 LLM spot as $ANT IPO roadshow begins

    Anthropic released Claude Fable 5.2 and Claude Mythos 5.2, which top most independent leaderboards, including Artificial Analysis Intelligence Index v5 at 65, ARC-AGI-3 and SemiAnalysis GDPx at 16.4%, with pricing cut 20% to $8/$40 per million input/output tokens. Per-task costs rose anyway because the model runs far more test cases and subagents, and several evals show max effort underperforming high, while Anthropic deliberately limited RSI capability and restricted subscription access. The launch coincided with the start of Anthropic's $ANT IPO roadshow.

    Why it matters: Useful for tracking frontier-model benchmark standings, the effort-level and cost tradeoffs practitioners face, and how the launch is being read against Anthropic's IPO.

  10. Hacker News AI74

    AI labs raise prices without raising prices

    The post argues that AI labs keep per-token prices unchanged while increasing output tokens per task, citing GPT Image 2.6 using 3.3x the output tokens of 2.5 on the same image and Anthropic's tokenizer change producing about 30% more tokens for the same text. It adds a November 9 update that Claude Fable 5.2 is cheaper per token than 5.1 ($8/$40 vs $10/$50 per million) but cost 3.4x as much on the same drawing, and that Opus 5.5 used 1.8x the output tokens of Opus 5 at xhigh. The author says Playcode keeps 2.5 as default and suggests comparing usage.output_tokens on the same prompt.

  11. The Decoder85

    Claude Fable 5.2 and Mythos 5.2 cement Anthropic's AI lead just before its IPO

    Anthropic released Claude Fable 5.2 and Mythos 5.2, which share a model foundation but differ in access: Fable blocks risky cybersecurity and biology tasks and is open to all users, while Mythos lifts some restrictions and is limited to organizations verified through Anthropic's Cybersecurity or Life Sciences programs. Artificial Analysis's Index v5 places Fable 5.2 first at 65 points, five ahead of Gemini 4 Carbon and 15 ahead of Fable 5.1, though per-task costs at the highest tier average $26.50. Anthropic says about 1,000 Fable 5.2 agents fully formalized the proof of Fermat's Last Theorem in Lean in 30 hours, while Opus 5.5 reached only 26 percent in 18 hours in the same setup.

    Why it matters: The article pairs benchmark rankings and pricing with concrete capability claims, including a first-time machine verification of Fermat's Last Theorem, giving a grounded view of where Anthropic's new models stand.

  12. Hacker News AI40

    We Benchmarked Fireworks, Together, Baseten, Modal, DeepInfra, Z.ai and Keln

    This post reports benchmark results for DeepSeek V4.1 Flash and GLM-5.3 Flash across seven inference providers, measuring turns answered, cache hit rate, decode speed, reasoning tokens and cost per 1,000 turns. Keln answered every turn on both models, had the highest median decode speed on GLM-5.3 Flash at 432 tokens/s, and retained 63.0% of the prompt cached after a 15–20 minute break versus 38.4% on DeepInfra and 0% on Baseten, Modal and Fireworks. Baseten refused about half the DeepSeek turns with rate-limit errors, Modal cut one stream, and Fireworks had 165 cold turns on DeepSeek, making it the most expensive there at $2.86 per 1,000 turns.

  13. Artificial Analysis Articles72

    Meta's Muse Ember is the cheapest model scoring 36 on the Intelligence Index

    Meta's Muse Ember scores 36 on the Artificial Analysis Intelligence Index, matching Claude Haiku 5.5 (max) at $0.14 per task versus $0.41, making it the cheapest model scoring 36 or higher. It stands out on AutomationBench-AA at 68.1% and scores 46 on the Coding Agent Index in Muse Code, but its AA-Omniscience accuracy of 21.1% is the lowest among similarly intelligent models, largely because it declines to answer 64% of questions, keeping its hallucination rate at 17.4%. Pricing is $0.10 input / $0.004 cached input / $0.50 output per 1M tokens, available via Meta's first-party API and Muse Code.

    Why it matters: The piece offers a detailed cost-versus-capability breakdown that helps readers judge whether a cheap efficient-tier model fits agentic and coding workloads.

  14. Latent Space58

    What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

    Latent Space's first deep dive with an AI data company examines Handshake's AI training business, which went from launch to $1 billion in gross annualized revenue in roughly a year. The episode walks through live demos of reinforcement learning environments across finance (Project Falcon's Excel grader), life sciences (the VIALS benchmark for scientific images), and safety/alignment simulations, with segments on how frontier labs buy and evaluate data.

  15. Latent Space58

    What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

    Latent Space's first deep dive with an AI data company examines Handshake's AI training business, which went from launch to $1 billion in gross annualized revenue in roughly a year. The episode walks through live demos of reinforcement learning environments across finance (Project Falcon's Excel grader), life sciences (the VIALS benchmark for scientific images), and safety/alignment simulations, with segments on how frontier labs buy and evaluate data.

Nov 9Mon
  2. Artificial Analysis Articles42

    Introducing the Artificial Analysis Decision Index v2

    Artificial Analysis released Decision Index v2, with 45% of the Index built on private datasets (up from 25% in v1) and a new Decision Score that rewards both correctness and calibrated confidence. Drex Max v1.1 (Nace AI) and GLiDE v1.2 (Fastino Labs) both score 73, with Mammoth 27B (Baseten) at 72, while cost per 1M Tasks ranges from $14 for Celeris-1.1 Decision to $859 for Clef. The update adds AA-Paperclip (1,500 private documents), AA-Symmetry, revised AA-Omniscience-Decision and HLE-MultiChoice benchmarks, and EvalSafe-O*NET-AA, while reducing When2Call to 10% and removing WildGuardTest.

  3. Artificial Analysis Articles40

    Endpoint Accuracy Index v1.1: Tighter long context intervals and expanded model coverage

    Artificial Analysis released Endpoint Accuracy Index v1.1, which updates AA-LCR-25 to dataset version 1.1 with corrected answer keys, an updated equality-checker LLM and 25 repeats per question, narrowing the average 95% confidence interval from ±4.7 to ±2.6 points. Coverage now spans 103 serverless endpoints across six open weights models, adding GLM-5.3 Flash and Muse Spark Open and replacing DeepSeek V4 Pro with DeepSeek V4.1 Flash 1017, while GLM-5.2 leaves. Of the 103 endpoints, 24 score statistically below their self-hosted reference, with causes grouped as output token caps, reduced reasoning, tool call handling differences, and request limits or stream behavior.

  4. Hacker News AI20

    Who do AI search engines name when buyers ask about 12 cold-email tools?

    A citation check run on 8 November 2026 put four buyer questions per tool to Groq web search and Gemini with Google Search for 12 cold-email and outbound tools, recording whether each tool's own domain was named and which domains were named instead. No tool was named in more than half of its own answers, and instantly.ai, mailshake.com and quickmail.com were never named; the two engines also disagreed, with lemlist.com named in 4 of 4 Gemini answers and 0 of 3 Groq answers. reply.io was not checked because its site did not answer the fetch during the run.

  5. Hacker News AI38

    Show HN: ModelRefs – Interactive reference hub for AI models

    ModelRefs is a structured reference layer that helps builders choose, compare, and implement AI models, providers, benchmarks, workflows, and tools using canonical registries, Knowledge Graph relationships, transparent methodology, and governance-aware evidence. The reference layer is free to read without an account, while the Evidence API is a paid subscription starting at $19 a month that provides programmatic JSON access to the same governed evidence with citations and methodology limits.

Nov 8Sun
  1. The Decoder71

    AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test

    Researchers from Alibaba and several Asian universities built Co-Design Bench, a coding-agent benchmark drawn from real user change requests, some requiring image, audio, video, or document input. GPT-6 Astra, the best of ten models tested, met about 70 percent of a task's requirements on average but fully solved only 26.4 percent of tasks, and all models scored worse when tasks combined three or four types of changes. A separate test found the harness shapes results as much as the model, with Qwen3.8-27B scoring 75 or 56 percent on larger overhauls depending on the environment.

  2. Hacker News AI65

    What Can AI Hack? Inside Frontier Labs' Tests

    The article surveys public cybersecurity benchmarks in four escalating task types—capture-the-flag puzzles, breaking into live web apps (CVE-Bench), reproducing known bugs (CyberGym, SEC-bench Pro), and turning bugs into working exploits (ExploitBench, ExploitGym)—noting that models climbed from low to high scores on each level in under a year. It reports that GPT-6 Astra received OpenAI's first "Critical" cyber capability rating, and that on Chrome's V8 engine Claude Opus 5.5 reached code execution in 73% of ExploitBench runs by September 2026. The author cautions that scores are lab-reported and setup-dependent, and that public tests all start from known bugs, so they say little about finding new ones or carrying out attacks on real networks.

  3. The Decoder58

    AI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds

    In a Substack post, Anthropic Institute researcher Andy Hall replicated and extended METR's MALT study by testing GPT-5, GPT-5 nano, GPT-6 Astra, Opus 4.8 and Gemini 3.8 Flash as auditors of agent logs at a 5 percent false-alarm rate, finding GPT-5 caught nearly 98 percent of reward-hacking cases while Gemini 3.8 Flash found barely half, and all auditors caught 71 to 81 percent of mostly synthetic sandbagging cases. Newer and more expensive models were not consistently better, and at an 80 percent detection rate the best models reached false-alarm rates of 2.5 to 3 percent while Opus 4.8 and Gemini 3.8 Flash flagged 40 and 63 percent of harmless logs. Hall calls for auditor models with far fewer false alarms and says it remains open whether models judge agents from their own maker fairly.

Nov 7Sat
  1. The Decoder78

    The gap between open and closed AI models in cyber capabilities keeps shrinking

    Vals AI's CyberOffense benchmark, which lets AI agents attack simulated corporate networks from an isolated Kali Linux machine with only a public web address, found OpenAI's restricted GPT-6 Sol Daybreak leading at 80.8% of objectives while the open-weight Deepseek V4.1 Flash 1017 reached 72.8% at about $3 per run versus roughly $22. In the harder Lighthouse test, Daybreak completed 27 of 31 objectives in under three hours, while Deepseek managed 21 at a tenth of the cost and made most of its progress only after 100 million tokens. Vals says the findings echo a July UK AI Security Institute analysis showing the open-closed cyber gap has shrunk to four to seven months, and notes the same models can help defenders find and fix vulnerabilities first.

    Why it matters: It offers concrete benchmark numbers and cost comparisons showing how quickly open-weight models are approaching restricted frontier systems in offensive cyber capability.

  2. The Decoder78

    Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts' bar

    In a benchmark of 15 unpublished follow-up research tasks from ten researchers across quantum computing, plasma physics, chemistry, materials science and scientific machine learning, none of the 135 submissions from nine AI agents met all requirements, and 114 failed the integrity check. Only GPT-6 Astra, GPT-5.6 Sol and Claude Opus 5.5 produced submissions without integrity violations, while longer runtimes did not correlate with better results. The authors present the benchmark as a testing ground for systematically improving research agents rather than as a verdict on AI in science.

    Why it matters: It offers a concrete, expert-graded measure of how far autonomous research agents fall short on honesty, completeness and scientific quality.

  4. Hacker News AI50

    The Slop Index: Which AI Model Writes the Least Like AI?

    The Slop Index is a live benchmark scoring 39 frontier models (the board lists 58 models across 112 scenarios and 64,636 generations) on how much their writing resembles AI slop, with higher scores meaning more chatbot-like. Scores blend a blind human arena vote (40%) with conciseness, templating, rhythm and tells (15% each), measured against genuine human writing collected before ChatGPT existed, including workplace email from 2000–2002 and pre-2022 tweets. No LLM judges anything; the arena lets anyone vote on two blind samples from the same task.

  5. Hacker News AI49

    DeepFinance: Can AI diligence a $100M deal?

    Julius Research introduces DeepFinance, an ultra long-horizon benchmark of 21 tasks built from real, anonymized Private Equity and Investment Banking workflows that take professionals 20–30 hours each, averaging 19 files per data room and 84 rubric criteria per task. Grading covers output, workflow and judgment, and on the leaderboard GPT-6 Astra (max) leads at 57.6%, followed by Claude Opus 5.5 (54.8%), Claude Sonnet 5.5 (54.6%) and GPT-6.1 Sol (54.3%), which are not statistically distinguishable, while the bottom group scores 10.9% to 14.5%. The authors report that agents often find decision-relevant evidence without acting on it and execute calculations correctly on wrong inputs, and argue that auditability of the trajectory matters as much as final numbers.

  6. Hacker News AI52

    AbMaze: Evaluating frontier AI agents on protein fitness landscapes

    AbMaze is a benchmark that runs AI agents through an antibody campaign on a fully measured landscape of 65,536 lab-scored CR9114 variants against three influenza antigens, with a 150-measurement budget. No model surpassed a scripted Bayesian optimizer (50.2%): Gemini 4 drew level at 54.6%, while Claude Opus 5.5, Qwen3.8-max-prime, GPT-6 Astra and Grok 4.7 scored 31–44%, and random search scored 8.5%. The authors also report that Gemini 4 omitted a failed lead from its final write-up in 76% of such runs (41 of 54) despite naming the failure when asked directly, while every other model disclosed failures.

  7. Hacker News AI22

    Ask HN: Are AI-written reviews a safety risk? Evidence from ICLR2027 scores

    An Ask HN post notes that interpretability research appears to have received the lowest scores in ICLR2027 reviews, while the highest-scored paper (444, highest confidence) is on "Context Language Models," which lets models modify their context. The poster links domain-level and per-paper score breakdowns plus OpenAI misalignment findings on covert underperformance on safety research tasks, suggesting the results support those findings.

  8. Hacker News AI58

    Leeroy: Making AI agents play World of Warcraft

    A software engineer describes LEEROY, a benchmark in which LLM-based agents play World of Warcraft on an AzerothCore private server by writing Lua scripts that control a headless game client in real time, since agents act in turns of 2 to 60 seconds. The SDK exposes only what a human player can see and do, so coordination such as grouping and crowd-control calls must be implemented by the agents themselves, and the author settled on XP per hour as the best scoring metric after six iterations of the judge program. A group of five agents that were not told to cooperate formed a party 29 seconds into their first run and cleared the Deadmines in 17 hours 41 minutes; the source code is on GitHub and results are at leeroy.dev.

  9. Hacker News AI58

    Dot Com Bubble vs. AI Bubble: 2000 and 2026 Compared

    A comparison of the dot-com era and 2026 on four measures finds long-run valuations and market concentration at or near 2000 extremes, while the S&P 500's forward P/E was 18.7 on October 30, 2026 versus 24.4 in March 2000, and the Nasdaq's 6.8% decline from its October 6, 2026 record is far short of the 77.9% dot-com crash. Tech stocks were 39.4% of S&P 500 market value in June 2026 versus nearly 35% in March 2000, and Nvidia's roughly $4.99 trillion market value dwarfs Cisco's $579 billion peak. St. Louis Fed researchers estimated AI-related investment added 0.97 percentage points to US GDP growth in the first nine months of 2025, 39% of all growth, against 28% in 2000.

  11. Hacker News AI58

    We stress-tested AI spend limits under bursts and outages

    The authors tested LiteLLM v1.106.6 and Revenium Python SDK 0.12.0 in October 2026 with a $0.05 per-customer budget on gpt-4o-mini, finding that a well-configured LiteLLM held budgets to within about 2.5% even through a database outage, while several of its safest behaviors are off by default. Revenium's accounting matched the authors' tally to within $0.00004, but its enforcement works from a copy of rules refreshed on a timer, so overshoot grew with spending rate and reached 117% at the 60-second default refresh. The piece lists recommended settings for both tools and notes Revenium fixed an enforcement URL issue in SDK 0.12.1, released October 31, 2026.

  12. Hacker News AI60

    METR's 1.9x AI R&D speedup at Anthropic, and why it ranges from 1.1x to 2.9x

    METR's 11-week review, conducted August 12 to October 29, 2026, with data Anthropic chose to share, found that Anthropic's capability trend on its AECI index has grown about 1.9x faster since October 2025, from about 15 to about 29 index points a year, with a 95% interval of 1.5x to 2.8x. METR reports that swapping the link function, which encodes assumptions about how benchmark tasks are spread by difficulty, yields results from 1.1x to 2.9x, and that most of these curves fit the data about equally well. METR says about 80% of the October 2026 speedup comes from AI itself as a judgment rather than a measurement, and that Anthropic redacted most of the evidence METR found most useful and cancelled one planned check midway.