Tracking the past to predict the future