These were some magical results from distillation by @geoffreyhinton that really shocked me when I first saw them. The TLDR: distillation has incredible robustness wrt the training
Geoffrey Hinton's Distillation Results
Geoffrey Hinton's distillation results are shocking, showing incredible robustness in training data.