Best practice for queuing experiments on code changes

Questions
1

Dear community,

Thank you very much for DVC and the new features from the version 2!

What would be your recommendation on how to queue experiments which depend on code changes?

The use case is to queue as individual experiment each change of the content of a Python file declared as a deps of a DVC stage.

For example:

  1. one is on a branch,
  2. makes a change on a Python file,
  3. do dvc exp run --queue for a DVC stage depending on this Python file,
  4. then do another change on the same Python file,
  5. then do dvc exp run --queue for the same DVC stage,
  6. and then do dvc exp run --run-all.

It seems that committing each change is an anti-pattern and not very usable.

But if the change are not committed, git shows the changed file hanging around as modified and the changes corresponding to an experiment seem then not tracked.

The idea would be to replicate what is below but when the changes are not on a params.yaml file:

Thank you.

2

Hi @pafonta, dvc exp run (with or without --queue) should work out of the box to generate experiments with your code changes. The feature is designed to work with any changes in your DVC/git repo, it is not specific to parameters (experimenting with modified parameters is just the common use case).

But if the change are not committed, git shows the changed file hanging around as modified and the changes corresponding to an experiment seem then not tracked.

Once you have run the experiments, you don’t need to keep the code changes in your workspace (and you do not need to commit them anywhere yourself). They will still be tracked as a part of the DVC experiment.

If/when you decide you want to keep one of your experiments, you can use dvc exp apply to re-apply the code changes back to your workspace.

The code changes for an experiment can also be retrieved directly in git - under the hood, experiments are essentially custom git branches. If you use dvc exp show --sha, you can see the git SHA for your experiments, and that SHA can be used in commands like git diff.

3

Hi @pmrowla,

Thank you very much for your prompt and detailed reply!

This helps me and solves my issue.