Dvc metrics diff on metrics stored in dvc-cache?

Questions
1

Hello!

Does the dvc metrics diff ref_a ref_b command work on metrics which are tracked not by git, but instead by dvc?

Our current setup is the following:

  • We have a dvc-pipeline-stage, which produces a eval.json, which is cached by dvc
    dvc run -d dep1 -d dep2 -m eval.json -f data/evaluation.dvc python my_script.py
    • According to the docs, the lowercase -m puts eval.json on the gitignore and into the dvc cache
  • Our model is trained remotely via a CI-Pipeline, which after a successful run executes a git add . && git commit && git push && dvc push
  • On our local machines then we execute a git pull && dvc pull. We now have the trained model and our eval.json locally available
  • When we now execute dvc metrics diff some_other_commit_hash the command does not find the eval.json for the some_other_commit_hash:
    $ dvc metrics diff 5f036012bc9d35b7240eab3b7a42792093612ba8 
WARNING: Metrics file 'data/evaluation/eval.json' does not exist at the reference '5f036012bc9d35b7240eab3b7a42792093612ba8'.
        Path               Metric          Value                Change      
dvc_pipeline/data/evaluation   f1       0.11549247896660152   diff not supported
/eval.json                                                                      
dvc_pipeline/data/evaluation   acc      0.6784822940826416    diff not supported
/eval.json                                                                      
dvc_pipeline/data/evaluation   loss     1.1063932177428895    diff not supported
/eval.json                                                                      
data/evaluation/eval.json      f1       0.43913782532829426   diff not supported
data/evaluation/eval.json      acc      0.750374436378479     diff not supported
data/evaluation/eval.json      loss     0.6777220140143865    diff not supported
  • I’m assuming this command fails, since (I’m guessing) dvc looks for the eval.json in the git-cache, but not in the dvc-cache (where it is stored)

Is there a flag, to tell dvc metrics diff ... that the metrics-file is stored in the dvc cache? Or is this expected to work, and we have an error somewhere else?

Thanks in advance!

2

Hi @rabefabi,

I’m assuming this command fails, since (I’m guessing) dvc looks for the eval.json in the git-cache, but not in the dvc-cache (where it is stored)

No, DVC should not be looking in Git for this metric (if it does, it would be a bug). It should just use the command hash as a reference to find the corresponding metric file in the DVC cache. But I think probably you’re just not pulling all the historic metrics into your local DVC cache — you could explore .dvc/cache to try and figure this out BTW but it’s kind of tricky knowing what the metrics file name should be.

  • On our local machines then we execute a git pull && dvc pull . We now have the trained model and our eval.json locally available

Try dvc pull --all-commits here instead. More details in pull

Please let me know if that helped and we’ll take it from there.

2 Likes
3

Thank you, you’re right and I did not pull the complete history, which is why the command failed.

For anyone else running into this issue:
I didn’t actually need my complete history since I switched to working with git tags.
To pull the data for all my tags I did the following:

$ dvc pull -T
$ dvc metrics diff my_first_tag my_second_tag

This ensures that all necessary data is pulled and can be found by the dvc metrics diff command.

Thanks again for the swift response, this forum helps a lot to find my way into dvc :slight_smile:

2 Likes