Then here (Data Versioning)
(I indicate this for reference, the error I got is related only to DVC)
In the DVC part I do
dvc init
and then when I got some data and my scripts produced some models (2 files in outputs)
dvc add data
dvc add outputs
later when the script was changed it produced a different model but I believe only one of the files changed
dvc commit -f
All other files were git managed
The problem starts here. I wanted to go back to the first models so I first did a
git checkout HEAD~
and then
dvc checkout
that is when I got the error
Building workspace index
Comparing indexes
ERROR; Can’t remove the following unsaved files without confirmation. Use --force to force
outputs/tfidf.joblib
Inside outputs there are two files : model,joblib and tidf.joblib and with the different scripts versions, it is model.joblib the one that changed.
But why does tfidf.joblib is considred “unsaved” if outputs was added to dvc???
Another warning I have is that for some reason when I dvc push–all-commits , all my git commits get
WARNING: failed to collect , skipping
but then the files get pushed.
I feel a bit nervous with these warnings
Is it a rule that we have to set up the dvc remote before doing all other dvc operations?
DVC is a meta versioning tool. It uses metadata that is saved in dvc.yaml and .dvc files to know what’s tracked. DVC is tracking the files that are specified on those files.
When you did git checkout HEAD~, I assume “that Git commit” did not have any of those files tracked, and as much as dvc is concerned, you don’t have such files being tracked. So, on dvc checkout, dvc is being careful about not losing your data and wants you to confirm what you want to do. It’s not an error, just an expected workflow.
DVC is a “data” versioning tool, so it’s better for dvc to be careful than end up deleting your data.
As the option suggests, --all-commits reads all your git commits. It might be that some commits were broken or had incomplete metadata. So, in that case, it’s okay to ignore this.
(--all-commits is not a everyday used option. The normal workflow is to just dvc push after you make any changes).
This depends on your usecase/workflow. remote acts as a backup location. All commands work without remote, and you can setup the remotes at your leisure.
Regarding dagshub tutorial, I’d suggest contacting them regarding the issues. The tutorial seems to be targeted towards older version of DVC, and is not being updated.
This tutorial was last updated to DVC version 1.10.1. If you are using an older version, please update. If you are using a newer version, be aware the behavior of some commands may change.
Alright but as you can see in the warning , it is only one of the two files in outputs that gets a warning about.
I have previously added the whole directory outputs to dvc so I assumed that both files would be managed.
Curiously after I do a force checkout now I can do checkouts normally…