Re-run changed final stage on previous model versions

Questions
1

Apologies if this has been asked before. Lets say I have a pipeline which finally outputs some metrics. I have used git tag to checkpoint several model versions. Then I decide I am not using the correct metric so change the final stage which generates the metrics. I would then like to run this updated final stage for previous model versions, without having to retrain those models. What would be the best way achieve this?

2

Hi @davesteps sorry for the delay.

Then I decide I am not using the correct metric so change the final stage which generates the metrics.

I’m understanding that your source code changes for the last stage, which generates metrics (let’s call it “evaluation”). Let’s say this latest commit is abc1234.

I would then like to run this updated final stage for previous model versions, without having to retrain those models.

This seems like a Git operation. You can checkout the previous tags and just apply the code changes to the evaluation stage (for example with git cherry-pick abc1234) and regenerate it with dvc repro evaluation.dvc.

I hope that helps. Feel free to continue to reach us here or in discord (dvc) — #need-help channel.

1 Like
3

Thanks @jorgeorpinel