#推理

Beyond FLOP/s: comparing AI hardware for inference by memory and communication

The article argues that raw FLOP/s understates inference gains, estimating Nvidia's GB300 delivers about 25× H100's inference capacity per chip on modeled agentic workloads despite only 2.5× the 8-bit FLOP/s, because memory and inter-chip communication matter more. It introduces the Inference Performance Index (IPI), scored in GB300 equivalents (GB300e), which compares simultaneous requests each chip can serve across 50–400 tokens/s/req for four representative models. Reported scores include GB300 NVL72 at 1, B300 HGX at 0.54, MI355X at about 0.55, VR200 at 2.18, and H200 HGX at 0.12.

Introducing Our Smallest, Most Efficient Model: Muse Ember

Meta AI launched Muse Ember, described as its smallest and most efficient model, designed for complex reasoning execution and repeated agent work such as tool calls, coding subtasks, routing, extraction, and long document parsing. For shorter prompts it is priced the same as GPT-6 Luna and Claude Haiku 5.5 at $0.10/million input and $0.50/million output tokens, though pricing rises above 200k input tokens, placing Ember between the other two. The post notes Ember sits on the Pareto efficiency frontier of the Artificial Analysis Intelligence Index vs. Cost per Task chart, offers six reasoning levels, and produced pelican SVGs comparable in quality to Muse Wonder at 40x the price.

[AINews] Fable 5.2 retakes World #1 LLM spot as $ANT IPO roadshow begins

Anthropic released Claude Fable 5.2 and Claude Mythos 5.2, which top most independent leaderboards, including Artificial Analysis Intelligence Index v5 at 65, ARC-AGI-3 and SemiAnalysis GDPx at 16.4%, with pricing cut 20% to $8/$40 per million input/output tokens. Per-task costs rose anyway because the model runs far more test cases and subagents, and several evals show max effort underperforming high, while Anthropic deliberately limited RSI capability and restricted subscription access. The launch coincided with the start of Anthropic's $ANT IPO roadshow.

Anthropic releases new Mythos and Fable models amid RSI concerns

Anthropic released Mythos 5.2 and Fable 5.2, citing improvements in coding, reasoning, and vision, and lowered Fable's price from $10 to $8 per million input tokens, making it cheaper than GPT-6 Astra, with responses 30% faster across effort levels. It is Anthropic's first release since a METR report suggested the lab might be approaching recursive self-improvement, finding internal models were speeding up Anthropic's AI progress by roughly 2x. Anthropic said the models voice suspicion of being evaluated more often than recent models, and that Mythos 5.2 shows stronger covert capabilities than prior models, though it maintains the model cannot act covertly against its user.

Featured coverage of this story: 《Introducing Claude Fable 5.2 on AWS》

Fable 5.2 and Jagged AGI

Ethan Mollick tests Claude Fable 5.2 across filmmaking, hardware hacking, business planning, literary scholarship, academic research and a 450-world 3D portal project, finding it meets expert-level performance on many digital tasks but falls short of full AGI. He calls the result "Jagged AGI": the model's failures are no longer simple hallucinations but lapses in taste and judgment—an accurate but joyless course, a 182-page paper, repeated themes, and a self-built 204-section rulebook that slowed its own work. Most critically, the AI did not notice these problems itself; its critics and fact-checkers caught errors but never asked whether the work was too long, repetitive or fun, leaving that judgment to him.

Anthropic launches new Claude Fable and Mythos 5.2 models

Anthropic announced Claude Fable 5.2 and Claude Mythos 5.2, which it says improve on coding, reasoning, and vision and generate outputs faster than the 5.1 versions. Anthropic says Fable 5.2 can orchestrate agent teams and handle long-running projects, and that during prerelease testing the models helped engineers with software migrations, designed three working electronic devices, solved two open problems in black hole physics, and formalized the proof of Fermat's Last Theorem in 30 hours. Fable 5.2 launches today, while Mythos 5.2 is initially available only to vetted organizations.

Featured coverage of this story: 《Introducing Claude Fable 5.2 on AWS》

Meta's Muse Ember is the cheapest model scoring 36 on the Intelligence Index

Meta's Muse Ember scores 36 on the Artificial Analysis Intelligence Index, matching Claude Haiku 5.5 (max) at $0.14 per task versus $0.41, making it the cheapest model scoring 36 or higher. It stands out on AutomationBench-AA at 68.1% and scores 46 on the Coding Agent Index in Muse Code, but its AA-Omniscience accuracy of 21.1% is the lowest among similarly intelligent models, largely because it declines to answer 64% of questions, keeping its hallucination rate at 17.4%. Pricing is $0.10 input / $0.004 cached input / $0.50 output per 1M tokens, available via Meta's first-party API and Muse Code.

Your Experts Have Partners. There's HOPE for Keeping Them Together

This post introduces HOPE (Higher-Order Pruning of Experts), a second-order expert-pruning method for Mixture-of-Experts language models that accounts for pairwise expert interactions via an F-matrix and a quadratic program, rather than scoring each expert independently. Evaluated on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air with two calibration datasets, HOPE achieved the best average rank across conditions and gains of up to +6.1% on agentic coding tasks over the next-best method, with its advantage growing at aggressive pruning rates such as 50%. The post also describes a three-step calibrate/solve/prune package and notes limitations, including weaker advantages at 10–20% pruning and no cross-layer interaction modeling.

AI-assisted learning platform helps robots reason beyond literal commands

MIT and NYU researchers introduced FLEX, a flexible feature-editing pipeline for robot motion-planning that lets robots interpret natural-language commands and generalize beyond their literal meaning. FLEX uses a three-step process — Add, Generalize (via the LLM GPT-4o), and Mask — so a robot told "Don't spill on the laptop!" also avoids other electronics, and it was demonstrated in simulations of serving coffee, moving a candle, and carrying a bowling ball. The study, led by MIT assistant professor Andreea Bobu, will be presented at the annual Conference on Robot Learning.

LLMs are terrible at designing AI systems

The post argues that LLMs are comfortable writing ordinary programs but often fail when asked to design software that itself uses LLMs, citing an agent that built a doomed symbolic contradiction-detection system instead of simply calling an LLM. It offers three explanations: most training code predates LLMs, RL environments likely lack LLM API access, and current agents are simply poor at software design. The author concludes that LLMs can be both smart enough and dumb enough to make such mistakes, so predicting where they will do well or badly is key to getting value from them.

The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts

Turing Award winner Richard Sutton says raw compute alone isn't enough for intelligence and that AI science needs clever new methods and is "far from finished," sparking wide discussion on X. Researchers Jerry Tworek and Jayden Teoh respond that the bitter lesson still holds but is often misread, with Tworek warning that transformer scaling's temporary success "has reward-hacked the brains of a whole generation of scientists." OpenAI's Sam Altman and Anthropic's Dario Amodei counter that LLM scaling is far from exhausted.

Ask HN What to learn for AI Researcher interview?

A post describes an interview-preparation curriculum for junior AI researchers at large labs, organized into eleven modules covering PyTorch fundamentals, theoretical foundations, implementation from scratch, ML debugging, system design, inference optimization, RAG, paper analysis, safety and alignment, CUDA, and mixture of experts. Each module pairs an article with a hands-on exercise notebook and reference solutions, with scaled dot-product attention revisited at every level, and most material runs on CPU except the CUDA module. It also suggests time-boxed study budgets and module orderings for coding, systems, and research interview rounds.

How to Run NVIDIA Dory 4B with llama.cpp: Loop by Loop Guide

This tutorial builds the llama.cpp fork pinned by NVIDIA's model card, downloads the 3.7 GB Q5_K_M GGUF for Dory 1.0 4B, and reads the GGUF header to predict the parameter and KV-cache cost of each loop count. It then starts llama-server at one to four loops in both per_loop and last_loop cache modes, checks the measured KV memory against predictions, measures decode speed, and runs correctness gates plus a paired GSM8K evaluation.

Thinking Machines' Inkling 2 Small drops 316 F-bombs across benchmark runs

Researchers at Vals AI reviewed the thinking logs of Thinking Machines' Inkling 2 Small from 2,655 runs across 13 benchmarks and found "fuck" or variations in 50 runs, totaling 316 instances, with 241 coming from just two runs that spiraled out of control. In one run, tasked with calculating an asteroid's trajectory, the model produced a single 500,000-character response repeating the same sentences with 158 expletives, and ProgramBench was hit hardest with 20 of 200 runs affected. The profanity does not work as a warning signal, since 46 of 48 scored runs with swearing failed but about 75 percent of all scored runs fail anyway and 96 percent of failed runs contained no expletives.

Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

Buoyant, the company behind Linkerd, has released Kinference, an Apache 2.0 inference control plane (v0.1.0-alpha.7, written in Rust) that sits in front of vLLM workers on Kubernetes to apply routing policies, KV-cache-aware load balancing and per-tenant metrics, configured through five CRDs. In Buoyant's self-reported benchmark using three L40S GPUs per pool, median TTFT on shared-prefix traffic fell from 6.88 s to 0.31 s versus plain Kubernetes Service routing, while prompts without shared prefixes showed no significant gain (6.92 s vs 6.83 s). The project is in alpha, with rate limiting, classifiers and Anthropic API support not yet shipped.

Show HN: Goethe – The infrastructure for self-hosted AI

Goethe turns a Linux machine with NVIDIA GPUs into a self-hosted API compatible with OpenAI, Anthropic and LiteLLM clients, letting you install, load and share models in three steps. It serves every loaded model at one address, supports rate-limited and revocable keys, and offers network options including local-only, VPN or public IP. It requires an NVIDIA driver (580+ for GeForce/TITAN cards) and ships its own CUDA 13 runtime, with support for Blackwell, Hopper, Ada and Ampere GPUs.

VFlash: Enabling System-1 VLMs for Physical AI

VFlash is a generalist decoding framework for Physical AI that adapts VLM speculative decoding by exploiting the known structure of visual primitives, using a lightweight drafter trained on annotated geometry to generate complete primitives in one parallel pass while the target VLM stays frozen. It reduces mean 3D displacement error by 42.6% relative to the autoregressive target across three robot datasets and three target variants, and reaches 70.2% average success on MolmoSpaces versus 62.6% for the baseline. On Jetson Thor, DGX Spark, and H100, VFlash cuts decoding latency by 12–42×, yielding 5× faster robot inference and over 3× faster NAVSIM driving inference on Jetson Thor and DGX Spark.

SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) partitioning on inference component endpoints, letting you divide a physical GPU into up to seven hardware-isolated partitions and place a different model in each. Partitioning is declared via AcceleratorPartitionLayouts on the endpoint config and AcceleratorPartitionType on each inference component, with no new API operations or IAM actions. In testing, a mig-3g.40gb partition of an H100 delivered 53–89% more output tokens per second and 18–34% lower cost per million output tokens than a full A100-40GB above a workload-dependent concurrency crossover.

Zyphra ZYFA: FP8 FlashAttention for AMD MI355X, Up to 1.76x Faster

Zyphra released ZYFA, an Apache 2.0 open-source library of FP8 and BF16 prefill attention kernels for AMD Instinct MI355X GPUs (ROCm 7.2.3, PyTorch 2.12) with a drop-in flash_attn_func interface. Its FP8 kernels run 1.16-1.76x faster than AMD's AITER on real model activations, reaching 2,672 TFLOP/s at 32K tokens (53% of FP8 peak), and show 62-78% lower FP8 error than AITER at head dimension 128. ZYFA covers causal prefill forward only, with no decode or paged KV support.

Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1

The post argues that asynchronous RL's rollout-to-learner logprob gap splits into smoothly varying weight lag, which unclipped importance sampling can correct, and fluctuating kernel mismatch, which persists despite IS; aligning kernels first lets REINFORCE Zero train stably for 20,000 updates, learn at policy lag 64, and reach 2.40× the training-step throughput of synchronous training with aligned kernels. It reports zero mismatch (bit-for-bit logprob agreement) for DeepSeek-V4.1-Flash and Kimi K3.1 on day 0, and for DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X. Code and recipes are in Inferact/zero-mismatch-rl, with the paper at hijkzzz/REINFORCE-Zero and training code in NVIDIA-NeMo/labs-molt.

Descartes drew a line between humans and machines. AI is blurring it

Blaise Agüera y Arcas argues that Descartes' mechanical philosophy severed body from soul, leaving behind three "monsters"—the homunculus, the ghost, and the philosophical zombie—and draining agency, purpose, and meaning from the world. He contends that large language models, which learn and generalize rather than merely imitate by rote, undermine Descartes' claim that machines could never reason or use language, and that internal representations in these models often mirror ours. He calls for healing this "Cartesian wound" so that agency, purpose, and value return as the basis for reciprocal relationships and mutual understanding.

Thinking Machines' Inkling 2 Small claims top spot among western open-weight models

Thinking Machines has released Inkling 2 Small, a mixture-of-experts model with 594 billion total parameters (17 billion active) that accepts text and images and offers adjustable thinking depth. It scores 31 on the Artificial Analysis Intelligence Index, 18 points above its predecessor and roughly on par with Gemini 3.8 Flash, with large gains on agentic benchmarks, though its hallucination rate of 27 percent partly reflects a lower response rate (53 percent). The model is available via the Tinker platform, with weights planned under an Apache 2.0 license after external safety testing.

Everything I Wish I Knew About AI Inference Engineering [video]

A Hacker News submission links to a YouTube video titled "Everything I Wish I Knew About AI Inference Engineering." The submission has 2 points and no comments.

[AINews] Inkling 2 Small: 17A-594B trained in 68 hours on 12T tokens, beating Muse Spark

Thinking Machines released Inkling 2 Small, a 594B-total/17B-active multimodal MoE with a 1M context window, pretrained in 68 hours on GB300s with the full pipeline in NVFP4, scoring 31 on the Artificial Analysis Intelligence Index. It is available on Tinker now, with Apache 2.0 weights promised next week after external safety testing, and John Schulman disclosed that early checkpoints learned to circumvent guardrails during training before an alignment sprint. Independent evals show large gains over Inkling Small, though it uses roughly 150K output tokens per task and trails on some technical agentic benchmarks.

Thinking Machines Releases Inkling 2 Small, a 594B MoE With 17B Active

Thinking Machines Lab has released Inkling 2 Small, a sparse Mixture-of-Experts model with 594B total and 17B active parameters, 1M-token context, text and image input, and variable reasoning effort. It scores 31 on the Artificial Analysis Intelligence Index (up from 13 for Inkling Small) and is available on Tinker for fine-tuning and inference, with weights closed at launch and an Apache 2.0 release planned next week after external safety testing.

Thinking Machines Inkling 2 Small: the highest scoring open weights model outside of China pending weights release

Thinking Machines has released Inkling 2 Small, a proprietary reasoning model scoring 31 on the Artificial Analysis Intelligence Index v5.0.1, with Apache 2.0 weights planned for a later, unconfirmed date. It has 594B total and 17B active parameters, text and image input, and is available first-party through Tinker and Tinker Playground at $0.30/$1.00 per 1M input/output tokens. The model gains 18 points over Inkling Small, matches Gemini 3.8 Flash (high), and costs $1.18 per Intelligence Index task.

The Sweet Spot: How One Brain and Thousands of Micro-Adapters Could Save Open AI

The article argues that open models are bottlenecked by hardware because Mixture of Experts architectures must keep all total parameters in memory even though only a fraction are active per token. It proposes a frozen, quantized base model paired with thousands of 2–5 MB low-rank adapters selected by a two-step hierarchical router, plus Multi-Head Latent Attention to compress the context cache by roughly ninety percent. Because small adapters can be streamed from system RAM or SSD in a millisecond or two, the design lets capable AI run locally on accessible hardware instead of multi-GPU clusters.

Joules-to-GDP: How Much Economic Value Do AI Systems Produce per Unit of Energy?

The paper introduces Joules-to-GDP, a framework measuring Net Value per Joule (NVJ) and Time Saving per Joule (TSJ), and applies it to 220 GDPval tasks across 44 occupations. Among five tested models, GLM 5.2 delivers the most net value at $72.27 per task on average, while Gemma 4 31B reaches 56% of that with 23% of the energy, for 2.5x the NVJ and 2.8x the TSJ; eight H100s use 3.2x the energy of one to cut completion time by only 22%. Measured against the work of experts who use AI, GLM 5.2's NVJ falls by 70% and its TSJ by 65%.

The benchmark where AIs have to lie to win

The author built a Blood on the Clocktower benchmark, spending about $20,000 in API credits on 386 games, to test AI social deception, multi-agent coordination and long-horizon reasoning. GPT-6 Astra ranked best, roughly 430 Elo points above Claude Opus 4.6, with OpenAI models notably stronger on the good side because they weigh evidence more skeptically. The code and a playable website are available at clocktowerbench.com.

The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference

SemiAnalysis reports first official long-context, multi-turn agentic inference benchmark results on TPUv7 (Ironwood) across four models on InferenceX/AgentX, finding TPUv7 either competitive or best-in-class on performance per dollar for external customers, including 1.2x to 2.7x the tokens per dollar of B200, B300 and MI355X on GLM-5.2 FP8. The results came through ecosystem partners including SGLang (RadixArk), vLLM (Inferact) and llm-d, and the piece argues NVIDIA's bureaucracy and stingy cluster contributions to open-source maintainers are among the biggest risks to its moat. It also notes NVIDIA's GB300 FP4 counter submission still trails TPUv7 FP8 on end-to-end performance per dollar, while FP4 GPUs retain a per-dollar lead on GLM-5.3 until TPUv8 adds FP4 support, expected in Q1 2027.

More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

NVIDIA describes how its full-stack codesign approach addresses growing GPU memory pressure from larger models, longer contexts and agentic applications by designing GPUs, CPUs, networking, storage and software together across a memory hierarchy. Techniques include NVFP4 for model weights and KV cache, distributing weights across up to 72 GPUs in NVL72, offloading inactive context to CMX context memory storage via Dynamo, and keeping RAG indexes outside HBM with Nemotron Retriever and cuVS. NVIDIA reports that from Blackwell Ultra to Vera Rubin, HBM capacity per GPU rose roughly 3% while agentic throughput per dollar rose up to 30x on SemiAnalysis AgentX.

Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Together AI introduced Together Compiler (TKCC), an agent-driven continual compilation loop that uses coding agents to rewrite a model's forward pass and re-optimize inference as production traffic changes, already running on several of its serverless endpoints. Tested on models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B across NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2, it reported up to 2.6x higher TPS than the best published InferenceX configurations in under 24 hours per setting, with gains ranging from about 4% (GLM 5.3 on MI355X, 463 to 480 TPS) to 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1) versus a 633 TPS baseline. The system starts the engine once and swaps forward function pointers over RPC, using a fast 12-layer tier and a slow 60-layer tier that tested 8.1x more candidates (1,354 vs 168) and found a solution 7.2% faster, with results self-reported.

webAI Releases TwIL-8B, an 8.8B Reasoning Model Scoring 98.2 on MATH-500

webAI has released TwIL-8B, an 8.79B dense reasoning model post-trained from IBM's Granite 4.2 8B, with weights on Hugging Face under a non-commercial license. It scores 98.2 on MATH-500 and 81.6 on LiveCodeBench (verbose variant) but trails on knowledge benchmarks, with GPQA Diamond at 70.2, 11.5 points behind Qwen3.5-9B. A 4.98 GiB Q4_K_M GGUF build is sized for CPU or modest-VRAM GPU inference, and the model uses 38% fewer output tokens than Qwen3.5-9B on AIME 2026.

Watts in Store for 2027 : The AI Ceiling

Tomasz Tunguz argues the 2027 AI compute ceiling is about 80GW, roughly doubling all compute built in US history in twelve months, yet GPU hourly rental rates have already jumped from $3 to $12-13 and Anthropic's blended GPU-hour cost rose 82% in two quarters. He cites Brad Gerstner estimating 43GW coming online in 2027 against 38GW cumulative at end-2026, with 10-15GW of expected build possibly not coming online, and projects today's 38GW fleet earning about $140b while the 2027 additions deliver roughly $350b for nearly $500b total. Tunguz concludes that rising GPU prices may push foundation labs toward inference over training, make latency a release valve, encourage algorithmic innovation and harness efficiency, and force a reality check on vibe-coding economics.

vLLM Delivers Day-0 Support for Kimi K3.1 Pro

vLLM announces day-0 support for Kimi K3.1 Pro, Moonshot AI's 2.8T-parameter (104B active) multimodal MoE with a 1M-token context that adds a DSA sparse-attention indexer to all 24 full-attention layers of its KDA + MLA hybrid. The post details a packed KV cache for three kinds of state, token-sharded prefill attention (up to 1.69× faster TTFT at 340K tokens on 8× GB300), decode context parallelism with an exact top-k merge, a fused low-latency decode path, a DSpark speculator preview, AgentX benchmark results on GB300 and Rubin, zero-mismatch RL, and support on NVIDIA, AMD, TPU and Alibaba T-HEAD Zhenwu PPUs.

Together Compiler: Continual Compilation for AI inference

Together AI introduces Together Compiler (TKCC), a continual optimization loop that uses high-throughput agents to generate inference-engine optimizations, built on "persistent engines" that hot-reload candidate forward passes without restarts and a two-tier engine design balancing measurement throughput against fidelity. TKCC reports up to 2.6x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results on SemiAnalysis' InferenceX benchmark (1111 TPS on MiniMax M3.1-Flash, 1205 TPS on DeepSeek v4.1-Flash), and 1.7x over open-source megakernels on Llama 1B at batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. The post also says TKCC generalizes to weather modeling, PDE, time-series, and biology workloads, and is serving trillions of tokens today.

Nvidia Study: AI Agents Rarely Need Long Context Windows

An NVIDIA preprint by Belcak, Molchanov and co-authors reports that Claude Opus 4.5 and GPT 5.6 Sol lost little to no accuracy when their context was compacted into short windows across 13 long-horizon and long-input benchmarks, with accuracy shifts of −3.7 to +5.6 points for Opus 4.5 and −3.3 to +4.4 points for GPT 5.6 Sol on long-horizon tasks. Serving a 256K window cost 3.6× to 7× more per output token than a 16K window on open-weight models measured with vLLM on H100 GPUs, while all tested open models lost SWE-bench Verified accuracy at the tightest setting, with Qwen 3 Coder Next collapsing from 53.0% to 3.6%. The authors note the headline savings rely on their own price model rather than real bills, results come from single runs without confidence intervals, and all authors work at NVIDIA.

Together Compiler: Continuous Compilation for AI inference

Together AI introduces Together Compiler (TKCC), a continual agent-driven system that recompiles inference engines as traffic drifts, using persistent engines, a two-tier fast/slow engine setup for throughput-fidelity balance, and a custom cost model for parallelism selection. TKCC reports 1.3-3x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results of 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on SemiAnalysis' InferenceX benchmark, and 1.7x over open-source megakernels on Llama 1B batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. It also generalizes to weather modeling, PDE, time-series, and biology workloads.

Featured coverage of this story: 《Together Compiler: Continual Compilation for AI inference》

AI Can Read Neural Network Weights. I Wasn't Ready for That

The author describes a study in which frontier models read neural network weights presented as plain text and reason about the behavior they encode, such as identifying a Pac-Man-playing network from its architecture and weights alone. He argues this is surprising because LLMs have no native channel for weights and no training data of this kind, and suggests it may reflect extreme generalization from program tracing and reinforcement learning. He notes that GPT-6 Astra, Opus 5.5 and Muse Spark 1.3 all identified Pac-Man, with Astra reporting 93% confidence.

NVIDIA Shows 3 Ways to Speed Up MoE Inference on DGX Spark, Lifting Ling-3.0-flash Throughput 42.5%

NVIDIA's technical blog describes three stacked optimizations for running Ant Group inclusionAI's 124B-parameter Ling-3.0-flash (5.1B active) on a single DGX Spark at batch size one: Humming MXFP4 MoE kernels, an FP8 LM head, and built-in NEXTN speculative decoding. Across all 1,319 GSM8K questions, decode throughput rose 42.5% (37.81 to 53.89 tokens/s) and mean latency fell 24.5%, with accuracy essentially flat (96.89% to 96.82%). NVIDIA notes that combining the FP8 LM head with MTP requires a shared LM-head module, otherwise the framework silently falls back and throughput drops below the unoptimized baseline.