TokenGate

TokenGate speeds up LLM prefill by up to 11.3x

TokenGate teaches LLMs to conditionally skim context tokens, achieving up to 11.3x faster prefill and drastically reducing peak memory for long-context tasks.