I’m running a command remotely through ssh, and it takes the remote cluster a while to finish the script. The problem is that DVC fails on this step because it immediately checks if the output file was created. Apart from just removing this file as an output, is there a way to have DVC wait and/or ignore whether the file was created?
Hello, @JedPhillips . I don’t know what script you are using, but I guess that you need some synchronization before finishing it. For example, setting a while loop until the outputs had been generated.
Thanks for the reply. That’s what I’ve ended up doing, just adding a while loop in the bash script that runs the cluster commands. It still creates some issues since it can take a while for the files to sync across mounted drives, but better than nothing. If you think of any canonical way to do it using dvc, definitely let me know.