Outgrowing Your Distractions: Knowing When To Let Them Go!