I notice that when I amend a git commit, the experiments that were based on that commit no longer show up with dvc exp show or dvc exp list. This sort of makes sense because the commit they were based on changed, but what is your recommendation for this situation? Should we avoid amending commits which have experiments based on them? Or is there a safe way to do it? How can we recover experiments which were lost in this way?
It would be best to avoid amending commits in this way, but if you have to amend the commit for some reason, the recommended option would be to re-run your experiments to be based on the amended commit, as described in this issue:
If re-running is not a feasible option due to training time or similar issues, it is technically possible to manually rebase the underlying experiment Git refs to depend on the amended commit. This is not recommended unless you are really sure you know what you are doing with Git and DVC, and it is important to note that this requires (in this order):
rebasing the original experiment so that the experiment commit is based on the amended commit
renaming the experiment Git ref to account for the amended commit
Again, this is not recommended, if you make a mistake you could potentially lose the original experiment entirely and/or put your git/dvc repo into a state where dvc exp commands will no longer work.
Given a scenario where I have amended the tip commit on main (so the result is 8b88009..., and the following experiment fated-tarp (which is derived from the old un-amended commit 435f03b...):
$ git --no-pager show main
commit 8b880092308997d5c16b4075c4dfd0c8caa732d8 (HEAD -> main)
$ dvc exp list -A
$ git rebase --onto main refs/exps/43/5f03bb04272e215d0b33a4a6d7f66faec56db5/fated-tarp
warning: skipped previously applied commit 8b88009
Successfully rebased and updated refs/heads/main.
Assuming you only amended the main commit message, and that this experiment derived from the original un-amended commit, there will not be any conflicts during this rebase. If there are conflicts here I would suggest using git rebase --abort and then re-run the experiment as explained earlier.
Rename the exp ref (delete the existing one and then add a new one)