TokenGate speeds up LLM prefill by up to 11.3x
TokenGate teaches LLMs to conditionally skim context tokens, achieving up to 11.3x faster prefill and drastically reducing peak memory for long-context tasks.
Inference
All AI Pulse updates tagged "Inference".
7 signalsUltrafast OpenAI inference now on Perplexity Computer
Perplexity Computer now supports ultrafast inference of OpenAI frontier models. It's very fast, but comes with a credit drain warning.
Agent traffic strains inference servers with repeated prefixes
Rohan Paul notes agent turns resend long near-identical prefixes and request short tool calls; Inferact’s Yifan Qiao discussed this Serve topic at DigitalOcean’s summit.
Microsoft Maia Chips to Become Primary Inference System
Satya Nadella announced Microsoft will make in-house Maia chips the primary inference system, scaling to gigawatts, reducing reliance on NVIDIA and replacing AMD chips.
Modal Shares AI-SQL Speedups Over 10x Faster Than vLLM
A new Modal blog post by Charles and Shreya covers their work speeding up AI-SQL queries — why AI-SQL, why high-throughput inference, how they beat vLLM by over 10x, and the future of inference engineering.
AI enters volume economy as token demand surges
Citing Goldman Sachs, the post argues AI is entering a volume economy: total token demand is projected to rise 18x from Dec. 2025 to Sept. 2026, while frontier-model demand grows more slowly.
Lenovo Launches New AI Inference Server ThinkSystem
Lenovo and Lisa Su introduced ThinkSystem, a new AI inference server based on AMD EPYC, designed for secure real-time data analysis.