Long-context LMs seem great at tasks like RULER, but what about tasks like "find every contradiction in the given corpus"? - Performance suffers, especially as the input corpus scales (even with in-domain fine-tuning, not just zero-shot) - It changes a lot of core modeling
Long-context models struggle to find corpus contradictions
sewon__min says long-context LMs do well on RULER-like tasks but suffer at finding every contradiction as corpora scale, even with in-domain fine-tuning.