Machine Learning Blog | ML@CMU | Carnegie Mellon University

Task construction as the control surface in continued pretraining. A construction rule \(c\) maps each unlabeled example \(x\sim\mathcal D_u\) into a self-supervised prediction problem \((x_c,y,m)\), such as one-hot next-token prediction in language modeling or paired views and targets in DINO-style vision SSL. Standard continued pretraining fixes this rule before training, ...