# Dealing with large datasets and file quotas

**URL:** https://discuss.dvc.org/t/dealing-with-large-datasets-and-file-quotas/1941
**Category:** Questions
**Created:** [February 4, 2024, 1:37pm UTC](https://discuss.dvc.org/t/dealing-with-large-datasets-and-file-quotas/1941 "2024-02-04T13:37:10Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![finlay\_morrison](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@finlay\_morrison](https://discuss.dvc.org/u/finlay_morrison)
#### Post date: [February 4, 2024, 1:37pm UTC](https://discuss.dvc.org/t/dealing-with-large-datasets-and-file-quotas/1941/1 "2024-02-04T13:37:10Z")

</div>

In the cluster I am using, my working directory at ‘~’ is limited to 200GB and 160K inodes. My dataset has a large number of files which surpasses this limit. For this reason, I am currently storing the dataset in a separate directory that the cluster provides ‘/scratch’, which is there for the purpose of providing a large storage space for non backed-up data.

Is there any way to deal with scenarios like this with DVC? Essentially I’m wondering if I can store the files in a directory in ‘/scratch’ but still be able to use DVC to add, push, or pull the data to a remote repository in a DVC project in my home directory. Thanks for any help you can provide!

---

<div class="post-metadata">

### Author: ![dberenbaum](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@dberenbaum](https://discuss.dvc.org/u/dberenbaum)
#### Post date: [February 5, 2024, 1:22pm UTC](https://discuss.dvc.org/t/dealing-with-large-datasets-and-file-quotas/1941/2 "2024-02-05T13:22:28Z")

</div>

You can set the DVC cache to that scratch dir and link files from there to your workspace. See [config](https://dvc.org/doc/command-reference/config#example-cache-config-options) for an example, and see [Large Dataset Optimization](https://dvc.org/doc/user-guide/data-management/large-dataset-optimization) for considerations about different link types.

---

<div class="post-metadata">

### Author: ![finlay\_morrison](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@finlay\_morrison](https://discuss.dvc.org/u/finlay_morrison)
#### Post date: [February 5, 2024, 1:34pm UTC](https://discuss.dvc.org/t/dealing-with-large-datasets-and-file-quotas/1941/3 "2024-02-05T13:34:36Z")

</div>

Thanks, this looks to be exactly what I am looking for!