One Attention Head Matters Across Five ICL Task Families
One attention head proves important across five in-context learning (ICL) task families. After an in-depth mechanistic analysis of addition ICL, the team generalized the findings to four more families spanning both arithmetic and other tasks.