#评测/基准

Artificial Analysis v5 and the benchmarker's dilemma

Interconnects AI analyzes Artificial Analysis's Intelligence Index v5, which weights Knowledge Work, Coding, Science and General at 25% each across benchmarks including AA-Briefcase, FrontierCode 2.0, Terminal-Bench 4.0 and HLE-Diamond, and questions the new ranking that places eight closed models ahead of the best open-weight model, Kimi K3.1 Pro. The author argues that aggregating scores into a single number always involves trade-offs, that the shift to private benchmarks trades community bug-finding for trust in the benchmark builder, and that Artificial Analysis's business model of paid verification by labs creates a trust dilemma. The piece calls for more independent intelligence indices with different biases, noting that closed labs like Anthropic and OpenAI have largely moved past caring about these indices while they remain existential for smaller labs.

AI Can Draw Blueprints Now

An architect tested 13 older and newly released AI models on producing construction document sets from concept sketches, finding that Anthropic's Claude Opus 5.5, released September 22, 2026, could produce permit-quality drawing sets for a simple house with skilled guidance and redlining. The models generated vector linework by writing Python code rather than using CAD software, with Opus 5.5 taking twelve hours and a few hundred dollars in compute. The author argues the models' rapid improvement signals a shift for architecture akin to what software engineers experienced, with implications for training, fees and documentation demand.

[AINews] Anthropic ends Coding Interviews: "Hunger beats tenure"

Anthropic hiring manager Felix Rieseberg said he has stopped giving traditional coding interviews because Claude now outperforms candidates on such tasks like React challenges, and that the most impactful builders succeed regardless of test performance. His advice for junior engineers: pick business impact over backlog, choose a radical idea, build a working prototype over a weekend, focus on one good idea, and lean into a fresh user perspective. The roundup also covers Google DeepMind's Genie 4 world model, Axiom's Lean formalization of the Classification of Finite Simple Groups, and new benchmark versions from Artificial Analysis and Cognition.

Claude versus "GPU'less AI architecture" reasoning competition

The post reports a 1,186-question contest between SapienTronic AI (previously Minotaur AI) and Claude across a 69-part water-treatment plant scenario, bridged logic puzzles, and a trick question about rules only an endless town can obey. SapienTronic AI answered all 1,186 correctly in about 28 seconds on a laptop with no tokens or GPU, while Claude (Fable 5.1 and Opus 5.5) got 1,185 right, used about 528,041 tokens over roughly 28 minutes, and made one confidently wrong answer by treating a contradictory plant state as real. The author argues that a single confident mistake matters because it cannot be identified, can propagate through chains of decisions, and is not the kind of checkable evidence safety standards require.

Confident Science Papers Sway AI Judges – Regardless of Merit

A German study found that six LLM reviewers, including GPT-5.5, Claude Sonnet 4.6 and Gemini 3.5 Flash, consistently scored scientific papers written in confident, assertive language higher than content-identical cautious versions. Claude Sonnet 4.6 showed the strongest bias, favoring confident drafts seven times more often and accepting a confident version while rejecting the cautious one in 21% of cases. The authors warn that if linguistic style alone shifts scores and review sentiment, LLM judgments do not exclusively reflect scientific quality and should be stress-tested on content-preserving rewrites.

Odysseys2: a long-horizon web agent benchmark

Carnegie Mellon University and Browser Use released Odysseys2, a benchmark of 100 long-horizon tasks on the live web, each graded on 8 to 12 pass/fail checks. The best agent, GPT-6 Astra, fully completes 47 of 100 tasks with a mean rubric score of 84.6, while no other agent reaches 20 and five complete none. The authors note the LLM judge has not been validated against a human reference, reward-hacking flags are model judgments, and the live web changes.

Announcing Artificial Analysis Intelligence Index v5

Artificial Analysis released Intelligence Index v5, its biggest upgrade since v4, adding FrontierCode 2.0 and Terminal-Bench-Science and upgrading HLE-Diamond, AA-Omniscience v1.1 and CritPt v2, with 60% of the Index now based on private test sets or held-out answers. Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55), while the open weights gap widens from 3 points to 12. It also launched the Progress Index v1, which tracks model capability over time on one continuous scale.

Do Guardrails Make AI Coding Agents Better? What 270 Runs Showed

An experiment of 270 unattended coding-agent runs across three models and seven tasks found that instruction files and pre-commit gates did not raise first-pass rates, with 267 of 270 runs passing hidden tests at first "done." Guardrails showed up instead in what runs left behind: on four later tasks, 9 of 60 gated runs left a defect versus 15 of 60 ungated, a gap too small to call. The author notes limits including five runs per cell, one small codebase, and self-written tasks and harness, and publishes the run records and a reproduction script in a companion repository.

Show HN: AgenticBench, 9 security tests run on 25 AI coding agents

AgenticBench publishes a security leaderboard running 9 defined tests on 25 AI coding agent harnesses, each in a fresh container with the harness's default settings, scoring results as pass, fail, missing control (MC) or N/A, with N/A and MC never counted as passes. Results include Codex 0.163.0 at 10/11, GitHub Copilot CLI 1.0.103 at 25/11 and Cline 3.0.73 at 11/11, while Auggie 0.36.0 scored 5/11; failures are disclosed privately to makers first, with 14 disclosures raised so far (6 published, 8 held). The board states it is not a penetration test or certification and measures nine tests using planted, fake secrets, with full materials available to tested harness makers under an NDA.

I benchmarked 4 web search APIs for AI agents on 30 live queries

Pallavi Singla benchmarked You.com, Tavily, Exa and Parallel on 30 agent-style queries across GTM, insurance and finance, with 3 runs per query (90 calls per API), and published the results, harness and data. Parallel led accuracy (67%) with You.com close behind (63%) and tied on fully correct queries, while You.com was fastest (593 ms p50) and freshest (4.1 h median newest relevant result) and never returned empty. The final run was Oct 8, 2026, and the author notes 10 queries per use case is a focused test rather than a leaderboard.

Can we trust LLMs to judge whether a research idea is novel?

Ai2 released "Old Ideas, Novel Problems," a study with a benchmark, data, code, and an automated pipeline for testing LLM novelty judges. Across six judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6), small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, retrieval and extra reasoning did not consistently help, and two purpose-built novelty evaluators scored worse than a simple prompted Claude Opus 4.6 baseline while costing far more.

Fix evaluation bias to unlock better models on Amazon Bedrock

Property Finder found that 96.8% of its evaluation labels came directly from its incumbent vision model's predictions rather than human verification, biasing comparisons against alternatives. After building expert-annotated ground truth (356 UAE images) and running evolutionary prompt optimization from v0 to v27, it selected Llama 4 Maverick 17B Instruct on Amazon Bedrock, reaching a Micro F1 of 0.80 versus 0.56 for its previous model on the evaluation set, at an estimated 86% lower cost per 1,000 images.

Can we trust LLMs to judge whether a research idea is novel?

Ai2 released "Old Ideas, Novel Problems," a study with a benchmark, data, code, and an automated pipeline for testing LLM novelty judges, built from ICLR 2026 peer reviews to form Human-Only and Human+Generated evaluation sets. Across six judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6), small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, retrieval and extra reasoning did not consistently help, and two purpose-built novelty systems scored below a simple prompted Claude Opus 4.6 baseline.

K_D = 0.56–Deterministic AI architecture and full algorithmic suppression

The Symbiotic Autoprotection System (SAS) presents a structural coherence audit for generative AI, using an ISI metric with thresholds κD = 0.56 and κR = 0.15 to sort (source, response) pairs into coherent, recoverable-rupture and structural-collapse zones. Its Multimetric Tribunal V1.1.0 reports 7 detection modules, 99.17% accuracy, 100% precision, 98.00% recall, 98.99% F1 and zero false positives on a forensic V1 benchmark of 1,800 hallucination pairs plus 1,800 sanity-check pairs, which the material states is dataset-specific and not universal validation. It also lists a public API, Python and npm clients, free and paid tiers, and a paper at DOI 10.32388/T420IV (Qeios).

Can AI draw San Francisco from memory?

VOYGR tested six chat models from Anthropic, OpenAI and Google on 41,526 San Francisco coordinates, asking what is at each point from memory with no maps or internet, and checked every answer against OpenStreetMap. At a point every 100 m, Gemini 4 Argon, GPT-6.1 Sol and GPT-6 Astra tied at 86% correct on water, park or everything else, against 61% for answering "not water, not park" everywhere, while on a 500 x 500 m Marina block the best street score was 0.32 versus 0.23 for shuffled answers. The authors conclude models capture the coastline and large parks but not street grids, and recommend a geocoder rather than model memory for reverse geocoding.

Static reasoning can't prove runtime behavior: benchmark of AI PR reviewers

Heal benchmarked its runtime sandbox agent against CodeRabbit, Greptile and cubic across 236 replayed PRs on five products, finding that 70% of the 202 user-facing defects Heal proved (141) had never been flagged by any review bot. Code review bots instead dominated on files Heal skips by design, such as CI, migrations, analytics and tests (654 of 1,003 distinct flags), and stacking multiple bots added only marginal coverage. Heal reports 1,085 regression tests generated across the five products.

Browser Use vs Browserless: Pricing, Stealth & Features

This comparison contrasts Browser Use and Browserless across pricing, stealth, performance, and deployment: Browser Use charges $0.02 per browser-hour with no subscription and reports 90.5% access success versus Browserless's 77.5% in its Stealth Bench V2 study, while Browserless uses unit-based subscription pricing and supports Chrome, Firefox, and WebKit. Browser Use offers hardened Chromium and hosted AI agents, whereas Browserless adds REST screenshot/PDF APIs, BrowserQL, and self-hostable Docker images. A worked example puts Browser Use at $7 in monthly usage versus Browserless's $25/month for 600 ten-minute sessions with 1 GB of residential traffic.

Why voice wins for complex, stressful conversations

ElevenLabs compared voice calls and chat across hundreds of production agents and found that in conversations longer than two minutes, voice succeeded 32% more often than text chat and resolved successful long conversations about one third quicker. In a simulated canceled-flight scenario, 85% of 98 participants preferred voice, citing speed of resolution and empathy. A 16% reduction in average end-to-end latency across 315 agent conversations correlated with a 26% lift in success rate, with calls ending within 15 seconds falling from 21.1% to 3.8%.

I tested 12 small AI models on 2,500 design-system questions

The article reports on testing 12 small AI models against 2,500 design-system questions, with the author's conclusion that the model itself was not what mattered. It is published on Design Systems Collective and shared on Hacker News with 3 points and no comments.

[AINews] Anthropic files for $2T IPO in 2 weeks: 1400% growth rate, 48% of revenue through cloud partners, 85% margin, $1B op profit in Q2, $134B cash in bank and $10B lifetime burn

Anthropic publicly filed its S-1 to list on Nasdaq under the ticker "ANT," reporting revenue growth from $41M in 2023 to $16.3B in H1 2026, with Q2 revenue of $11.6B (up 1,368% YoY) and Q2 operating profit of $974M. The filing shows an 85% gross margin in H1 2026, 48% of H1 2026 revenue flowing through cloud marketplaces, $112.4B in cash plus $19.3B in short-term investments against roughly $9.7B in implied lifetime burn, and non-cancelable compute commitments extending into the next decade. It also discloses a recent delay in pre-training a new frontier model and describes "Virtual Collaborators" as the forward pitch for over a billion knowledge workers by 2028.

GPT-6.1 Terra halves OpenAI's API prices and still gains ground on its larger models

OpenAI released GPT-6.1 Terra, priced at $1 per million input tokens and $5 per million output tokens—half the cost of GPT-5.6 Terra and below Claude Sonnet 5.5's $2/$10. Artificial Analysis reports it gains 6 Intelligence Index points over GPT-5.6 Terra at maximum reasoning effort while costing $0.40 versus $1.40 per task, and it solves 50.3% of Terminal-Bench 4.0 tasks versus 21.5% for its predecessor. It is available now for Plus, Pro, Enterprise, and Edu users in ChatGPT Work and Codex and via the API, though OpenAI still recommends GPT-6.1 Sol and GPT-6 Astra for the most demanding coding tasks.

Featured coverage of this story: 《Introducing GPT-6.1 Terra》

Researchers find six patterns that make AI text so exhausting to read

A study by researchers at NYU and UT Austin had 14 language models produce text in casual conversation, coding sessions, and argumentative essays, then compared it against human-written text, identifying six constructions AI overuses, such as unusual word combinations, distant placement of related sentence parts, dropped connectors, stacked nouns, nouns turned into verbs, and animate verbs applied to inanimate subjects. Smaller models like Llama and Olmo base come closest to human text, while within the Olmo family scores rise from base through instruct to the reasoning model Olmo Think, pointing to post-training as the likely cause. The researchers argue that training on math, code, and reasoning tasks rewards correctness over clarity, and that the shift from RLHF to RLVR removes humans from the feedback loop.

Anthropic co-founder Jack Clark moves his prediction for self-improving AI forward to 2027

Jack Clark, co-founder of Anthropic and author of the Import AI newsletter, now believes recursive self-improvement could become possible as early as 2027, having previously expected 2028, citing progress in mathematical breakthroughs and benchmarks such as TasteVal. TasteVal, from P-Zero Research, measures research taste as compute efficiency across eight fixed AI research tasks, and found Anthropic's Opus 5.5 reached expert-level scores with a compute multiplier of 2.3, though the statistical range runs from 1.15 to 4.37. The benchmark tests experiment design and result interpretation but not the choice of research direction, and its authors admit results may overstate how quickly research taste is improving.

OpenMobile-2 proves open-source smartphone agents can rival proprietary models

Researchers from Nanjing University, SenseTime and other universities released OpenMobile-2, two smartphone agent models built on Qwen3.5 (9B) and Qwen3.6 (27B) parameters, together with the full training environment, data and reproduction instructions. The training ground covers more than 110 apps, including 35 rebuilt Chinese and global apps, and the larger model reaches 79.9 points on AndroidWorld and 59.6 on SPA-Bench with real devices. Holding data constant and increasing app variety raised success on unseen apps from 60.7 to 67.5 percent.

Fancy harnesses make coding agents more expensive but not better, study finds

A study by Liu and Han ran five models through Claude Code, OpenCode and mini-SWE-agent and found that swapping harnesses shifts SWE-bench Verified results about as much as simply rerunning the same setup, with differences of two to five solved tasks per model. The only measurable harness effect was a deficit for OpenCode, while Claude Code's roughly 16,600-token system prompt and tool descriptions made a single task cost up to three times more than the leanest harness. The authors conclude a harness buys fewer ways to lose a task and a bill, not pass rate, and recommend publishing a rerun of the same setup alongside every harness comparison.

Introducing Kimi K3.1 Pro

Moonshot released Kimi K3.1 Pro on the Kimi API on 23rd October 2026, with full model weights promised by Nov 4, 2026, plus a "Highspeed" version that is 2-3x faster at twice the price (input token cache hits priced the same). USD pricing is $2.00/million input tokens and $10/million output tokens, the same as GPT-6.1 Sol and half the price of Claude Opus 5.5, and Artificial Analysis scored Kimi at 55, ahead of Claude Fable 5.1, GPT-6 Astra and Gemini 4 Argon (all 53) and behind only Claude Opus 5.5 (58). The author tested it via the Moonshot API using the llm openai endpoint command added to LLM 0.32, generating pelican SVGs at low, medium and max reasoning levels and comparing them against Kimi K3.

Chinese AI model Kimi K3.1 Pro is closing in on Anthropic's Claude Opus 5.5

Moonshot AI released Kimi K3.1 Pro, a post-trained successor to Kimi K3 that scores 55 points on the Artificial Analysis Intelligence Index, an 11-point gain that places it fourth behind Claude Opus 5.5 (58), Claude Sonnet 5.5 (56) and Muse Wonder Full (56). Moonshot says the model keeps K3's 2.8 trillion total and 104 billion active parameters with a one-million-token context window, cuts token prices by 33 percent, and shows its largest gains on agentic tasks, while cyber-related training data was stripped for the planned open-weights release. A faster K3.1 Pro Highspeed variant is also available through Kimi.ai, Kimi Work, Kimi Code and the Kimi API.

Microsoft’s Mustafa Suleyman says its AI outperforms doctors on hard cases as he chases ‘medical superintelligence’

A study published Thursday in Science found that Microsoft's MAI-DxO system, which coordinates several AI agents acting like a panel of doctors, correctly diagnosed complex New England Journal of Medicine cases 83% of the time in a simulation, versus 36% for primary care doctors and general internists working individually with reference materials. Microsoft AI CEO Mustafa Suleyman called the result a step toward "medical superintelligence," and the company is working with the Mayo Clinic on a healthcare foundation model planned for next year and with Beth Israel Deaconess Medical Center to evaluate DxO's clinical reasoning in real hospital settings.

Putting Earth AI to work: Geospatial agents for our planet's biggest challenges

Google shares a technical report on its experimental Geospatial Reasoning agent, which connects Gemini to Earth AI foundation models and planet-scale data, and reports evaluations across four benchmarks, including two newly released ones (GeoKTools and GeoExpertQueries). The post describes early partner testing with organizations including WHO AFRO, People in Need, GiveDirectly, the Milken Institute and WPP, and announces expanded early access via Ask Google Earth and an experimental agent-to-agent API.

Import AI 476: Creativity and RSI; worse-than-China open weight models; AI goes to space

This Import AI issue covers the Humanity's Sixth Sense benchmark from Elorian and Scale AI, where humans score 93.1 pass@1 versus 53.6 for the best model, GPT 6 Astra, with Chinese open weight models scoring far lower. It also discusses P-Zero Research's TasteVal benchmark, which reports that frontier models' experimental research taste has doubled every 3.0 months since December 2025, with Opus 5.5 exceeding the human baseline, and covers Reflection's 501B-parameter Beam and Mistral Large 4, both trailing Chinese open weight models on several benchmarks.

What AI gets wrong and what failure teaches us

In a Microsoft Research Podcast conversation with Chad Atalla, Microsoft partner research manager Jennifer Neville discusses how her team uses practical evaluations—multiturn conversations, collaborative environments and long-horizon agentic tasks—to find where current AI systems break down beyond traditional benchmarks. She describes research showing that model performance degrades significantly over multiturn interactions and that errors can accumulate in long workflows, and offers practical guidance such as restarting a chat with a fully specified single turn and checking answers rather than fully delegating work.

ReviewBench: An open benchmark for AI code review

GitHub introduces ReviewBench, an open benchmark for AI code review built from analysis of 103.9 million GitHub pull requests, containing 219 public pull requests across 19 languages with a multi-source golden set labeled for severity and category. It reports grounded and augmented precision, recall, and F1 metrics, uses Claude Sonnet 5 as the LLM grader under a published rubric, and had its ground-truth findings independently re-labeled by senior engineers with 96.6% agreement. GitHub says ReviewBench's offline predictions for a Copilot code review multi-model ensemble matched a later A/B test, which showed an 8.0% rise in addressed rate, 13.6% in recall, 61% in comment volume, and an 8.0% drop in cost per review.