Upgrade to .60 produces .dvc file dependency error -- breaks pipeline

Questions
#1

I just upgraded to dvc 0.60.0 and when attempting to run the same commands I have run for ages I get the following error

Command run:
dvc run -f entropy_filter.dvc -d entropy_config.json -d ../../../scripts/entropy_filter.py -O junk_clusters.txt python ../../../scripts/entropy_filter.py entropy_config.json

output
ERROR: run_model.dvc cannot be a dependency

Note that the dependency to entropy_filter.dvc stage (entropy_config.json) is produced by run_model.dvc.

I then attempted to rerun the base command to regenerate run_model.dvc only to get the exact same error, that run_model.dvc cannot be a dependency. I then attempted to remove run_model.dvc and run the command again only to be given the same error. Finally I attempted a dvc gc which again gave the same error…

Only way to fix was to downgrade back to 59.2. Any suggestions on how I can change my pipeline to work with 0.60.0?

#2

Hi @jwelch !

It seems like you have some dvcfile in your project that uses run_model.dvc as a direct dependency. Could you run grep run_model.dvc -R ./ | grep "path:" in the root of your project and show us the output?

NOTE: we do need to improve that error message so it says which stage it belongs to.

#3

Hi @kupruser,

Thanks for the reply. Here is the output of the command when run on both run_model.dvc and entropy_filter.dvc

Run on run_model.dvc

  path: ../../../scripts/ufcpc_main.py
  path: config.json
  path: ../../../data/all/level3filter/R50_wobt_level3_good_less-116-117-64.ds
  path: entropy_config.json

Run on entropy_filter.dvc

  path: entropy_config.json
  path: ../../../scripts/entropy_filter.py
  path: junk_clusters.txt
#4

@jwelch i think there is a misunderstanding. I was asking to run this specific command from your project’s root

grep run_model.dvc -R ./ | grep "path:"

it will go through your project looking for run_model.dvc being specified as a dependency or an output in any other dvcfiles. Please run it an share the output.

#5

@kupruser Ah my bad I did indeed misunderstand. Indeed you were correct that there was an old dvc file that had a dependency on another dvc file. I thought I had removed all such references but apparently missed one. Thanks again!

1 Like
#6

Sure! Glad to hear it works now :slight_smile: Please let us know if you have any questions;