In the cluster I am using, my working directory at ‘~’ is limited to 200GB and 160K inodes. My dataset has a large number of files which surpasses this limit. For this reason, I am currently storing the dataset in a separate directory that the cluster provides ‘/scratch’, which is there for the purpose of providing a large storage space for non backed-up data.
Is there any way to deal with scenarios like this with DVC? Essentially I’m wondering if I can store the files in a directory in ‘/scratch’ but still be able to use DVC to add, push, or pull the data to a remote repository in a DVC project in my home directory. Thanks for any help you can provide!