# Dvc metrics diff on metrics stored in dvc-cache?

**URL:** <https://discuss.dvc.org/t/dvc-metrics-diff-on-metrics-stored-in-dvc-cache/386>\
**Category:** Questions\
**Created:** [May 15, 2020, 1:16pm UTC](https://discuss.dvc.org/t/dvc-metrics-diff-on-metrics-stored-in-dvc-cache/386 "2020-05-15T13:16:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rabefabi](https://avatars.discourse-cdn.com/v4/letter/r/74df32/32.png) [@rabefabi](https://discuss.dvc.org/u/rabefabi)\
**Post date:** [May 15, 2020, 1:16pm UTC](https://discuss.dvc.org/t/dvc-metrics-diff-on-metrics-stored-in-dvc-cache/386/1 "2020-05-15T13:16:41Z")

</div>

Hello!

Does the `dvc metrics diff ref_a ref_b` command work on metrics which are tracked _not_ by git, but instead by dvc?

Our current setup is the following:

- We have a dvc-pipeline-stage, which produces a `eval.json`, which is cached by dvc  
` dvc run -d dep1 -d dep2 -m eval.json -f data/evaluation.dvc python my_script.py`
  - According to the [docs](https://dvc.org/doc/command-reference/run#options), the lowercase `-m` puts `eval.json` on the gitignore and into the dvc cache

- Our model is trained remotely via a CI-Pipeline, which after a successful run executes a `git add . && git commit && git push && dvc push`
- On our local machines then we execute a `git pull && dvc pull`. We now have the trained model and our `eval.json` locally available
- When we now execute `dvc metrics diff some_other_commit_hash ` the command does not find the `eval.json` for the `some_other_commit_hash`:

```auto
$ dvc metrics diff 5f036012bc9d35b7240eab3b7a42792093612ba8 
WARNING: Metrics file 'data/evaluation/eval.json' does not exist at the reference '5f036012bc9d35b7240eab3b7a42792093612ba8'.
        Path Metric Value Change      
dvc_pipeline/data/evaluation f1 0.11549247896660152 diff not supported
/eval.json                                                                      
dvc_pipeline/data/evaluation acc 0.6784822940826416 diff not supported
/eval.json                                                                      
dvc_pipeline/data/evaluation loss 1.1063932177428895 diff not supported
/eval.json                                                                      
data/evaluation/eval.json f1 0.43913782532829426 diff not supported
data/evaluation/eval.json acc 0.750374436378479 diff not supported
data/evaluation/eval.json loss 0.6777220140143865 diff not supported

```

- I’m assuming this command fails, since (I’m guessing) dvc looks for the `eval.json` in the git-cache, but not in the dvc-cache (where it is stored)

Is there a flag, to tell `dvc metrics diff ...` that the metrics-file is stored in the dvc cache? Or is this expected to work, and we have an error somewhere else?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\
**Post date:** [May 15, 2020, 6:36pm UTC](https://discuss.dvc.org/t/dvc-metrics-diff-on-metrics-stored-in-dvc-cache/386/2 "2020-05-15T18:36:19Z")

</div>

Hi @rabefabi,

> I’m assuming this command fails, since (I’m guessing) dvc looks for the eval.json in the git-cache, but not in the dvc-cache (where it is stored)

No, DVC should not be looking in Git for this metric (if it does, it would be a bug). It should just use the command hash as a reference to find the corresponding metric file in the DVC cache. But I think probably you’re just not pulling all the historic metrics into your local DVC cache — you could explore .dvc/cache to try and figure this out BTW but it’s kind of tricky knowing what the metrics file name should be.

> - On our local machines then we execute a `git pull && dvc pull` . We now have the trained model and our `eval.json` locally available

Try `dvc pull --all-commits` here instead. More details in [pull](https://dvc.org/doc/command-reference/pull#options)

Please let me know if that helped and we’ll take it from there.

---

<div class="post-metadata">

**Author:** ![rabefabi](https://avatars.discourse-cdn.com/v4/letter/r/74df32/32.png) [@rabefabi](https://discuss.dvc.org/u/rabefabi)\
**Post date:** [May 18, 2020, 11:27am UTC](https://discuss.dvc.org/t/dvc-metrics-diff-on-metrics-stored-in-dvc-cache/386/3 "2020-05-18T11:27:35Z")

</div>

Thank you, you’re right and I did not pull the complete history, which is why the command failed.

For anyone else running into this issue:  
I didn’t actually need my complete history since I switched to working with git tags.  
To pull the data for all my tags I did the following:

```auto
$ dvc pull -T
$ dvc metrics diff my_first_tag my_second_tag

```

This ensures that all necessary data is pulled and can be found by the `dvc metrics diff` command.

Thanks again for the swift response, this forum helps a lot to find my way into dvc 🙂