Experiments go missing when commits are amended

Questions
1

Hello,

I notice that when I amend a git commit, the experiments that were based on that commit no longer show up with dvc exp show or dvc exp list. This sort of makes sense because the commit they were based on changed, but what is your recommendation for this situation? Should we avoid amending commits which have experiments based on them? Or is there a safe way to do it? How can we recover experiments which were lost in this way?

Thanks

2

It would be best to avoid amending commits in this way, but if you have to amend the commit for some reason, the recommended option would be to re-run your experiments to be based on the amended commit, as described in this issue:

If re-running is not a feasible option due to training time or similar issues, it is technically possible to manually rebase the underlying experiment Git refs to depend on the amended commit. This is not recommended unless you are really sure you know what you are doing with Git and DVC, and it is important to note that this requires (in this order):

  1. rebasing the original experiment so that the experiment commit is based on the amended commit
  2. renaming the experiment Git ref to account for the amended commit

Again, this is not recommended, if you make a mistake you could potentially lose the original experiment entirely and/or put your git/dvc repo into a state where dvc exp commands will no longer work.

Given a scenario where I have amended the tip commit on main (so the result is 8b88009..., and the following experiment fated-tarp (which is derived from the old un-amended commit 435f03b...):

$ git --no-pager show main
commit 8b880092308997d5c16b4075c4dfd0c8caa732d8 (HEAD -> main)

$ dvc exp list -A
435f03b:
        9746b08 [fated-tarp]

To rebase fated-tarp to be based on my new main:

  1. Get the full exp SHA and ref name for fated-tarp:
$ git show-ref -- fated-tarp
9746b08419c258dd4cab18a9a4e5c4f6e19c5881 refs/exps/43/5f03bb04272e215d0b33a4a6d7f66faec56db5/fated-tarp
  1. Rebase the experiment
$ git rebase --onto main refs/exps/43/5f03bb04272e215d0b33a4a6d7f66faec56db5/fated-tarp
warning: skipped previously applied commit 8b88009
Successfully rebased and updated refs/heads/main.

Assuming you only amended the main commit message, and that this experiment derived from the original un-amended commit, there will not be any conflicts during this rebase. If there are conflicts here I would suggest using git rebase --abort and then re-run the experiment as explained earlier.

  1. Rename the exp ref (delete the existing one and then add a new one)
$ git update-ref -d refs/exps/43/5f03bb04272e215d0b33a4a6d7f66faec56db5/fated-tarp
$ git update-ref refs/exps/8b/880092308997d5c16b4075c4dfd0c8caa732d8/fated-tarp 9746b08419c258dd4cab18a9a4e5c4f6e19c5881

Note that we changed the hash segment of the ref name to match the new amended main commit SHA, and that the experiment commit SHA 9746b08... does not change after the rebase.

  1. Verify that the rebase worked
$ dvc exp list
main:
        9746b08 [fated-tarp]
1 Like
3

Great this worked! Thanks for the detailed answer