Excited to share TokenGate. By teaching LLMs to conditionally skim context tokens, we achieve up to 11.3x faster prefill and drastically reduce peak memory for long-context tasks!
TokenGate speeds up LLM prefill by up to 11.3x
TokenGate teaches LLMs to conditionally skim context tokens, achieving up to 11.3x faster prefill and drastically reducing peak memory for long-context tasks.