# How to add data from "scratch" filesystem directly to "scratch" cache dir
**URL:** https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702
**Category:** Questions
**Created:** [March 15, 2021, 8:13pm UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702 "2021-03-15T20:13:44Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@marius](https://discuss.dvc.org/u/marius)
#### Post date: [March 15, 2021, 8:13pm UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/1 "2021-03-15T20:13:44Z")
My scenario is that I’m on a cluster with really limited home directory quota (40GB) but with TB of “scratch” space on a separate filesystem. I’d like to keep my Git repo on the home filesystem (say at `~/myrepo/`), but the DVC cache on the scratch file system (say at `/scratch/dvc`).
Its easy enough to configure the cache on the scratch filesystem with e.g. `dvc cache dir /scratch/dvc` and then use the `symlink` cache type so files just symlink to here.
But now, suppose I’ve generated a large file at `/scratch/newfile.dat`. How can I add it to the repo at a chosen path without it ever touching the home filesystem?
I would have thought:
```bash
dvc add -o ~/myrepo/newfile.dat /scratch/newfile.dat
```
would do it but that still seems to go through the home filesystem.
It seems if I add `--to-remote` it works as expected, but it _also_ (obviously) pushes to remote, which I don’t want.
Thanks for any suggestions.
---
### Author:  [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)
#### Post date: [March 15, 2021, 10:40pm UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/2 "2021-03-15T22:40:47Z")
Hi @marius,
> [@marius](#):
>
> Its easy enough to configure the cache on the scratch filesystem with e.g. `dvc cache dir /scratch/dvc` and then use the `symlink` cache type so files just symlink to here.
Glad you were able to find that info already 👍 That’s the first step indeed.
> [@marius](#):
>
> How can I add it to the repo at a chosen path without it ever touching the home filesystem?
>
> ```auto
> dvc add -o ~/myrepo/newfile.dat /scratch/newfile.dat
>
> ```
If you’re using DVC 2.x, then that exact command is what you need. It should transfer the data to the cache without needing much space available in the local system (it uses file chunking on as an intermediate step).
See also [add](https://dvc.org/doc/command-reference/add#example-transfer-to-the-cache)
Thanks.
---
### Author:  [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)
#### Post date: [March 15, 2021, 10:41pm UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/3 "2021-03-15T22:41:59Z")
> [@marius](#):
>
> ```auto
> dvc add -o ~/myrepo/newfile.dat /scratch/newfile.dat
>
> ```
Not sure about the absolute path you’re providing to `-o` though, please use a relative path from your project directory if needed, i.e.
```console
$ cd ~/myrepo
$ dvc add /scratch/newfile.dat -o newfile.dat
```
---
### Author:  [@marius](https://discuss.dvc.org/u/marius)
#### Post date: [March 15, 2021, 11:48pm UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/4 "2021-03-15T23:48:04Z")
> [@jorgeorpinel](#):
>
> If you’re using DVC 2.x, then that exact command is what you need. It should transfer the data to the cache without needing much space available in the local system (it uses file chunking on as an intermediate step).
Ahhh I see, that piece of info is key, thanks (and I am on 2.x). The reason I thought the home filesystem was being touched was because the write speed was kind of slow, indicative of having gone scratch → home → scratch, as compared to a straight scratch → scratch copy which on this system is much faster. So I suppose that’s just because the data _does_ pass through the home filesystem, but it does so in chunks so it shouldn’t overwhelm my quota. I guess a question I have is why not a direct copy, but in any case this is very helpful and solves my major worry about the quota. Thanks!
---
### Author:  [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)
#### Post date: [March 16, 2021, 2:20am UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/5 "2021-03-16T02:20:09Z")
> [@marius](#):
>
> as compared to a straight scratch → scratch copy which on this system is much faster
Yes that would be ideal. I’ll check with engineering on this… Thanks for the feedback!
---
### Author:  [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)
#### Post date: [March 16, 2021, 6:20am UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/6 "2021-03-16T06:20:14Z")
Another quick clarification, BTW: the transfer doesn’t actually use any local disk space at all, as the chunks are stored locally in memory only.
---
### Author:  [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)
#### Post date: [March 16, 2021, 6:31am UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/7 "2021-03-16T06:31:22Z")
> [@jorgeorpinel](#):
>
> that would be ideal. I’ll check with engineering
Finally, I did check and for now DVC can’t support direct transfers from external data locations to remote storage (without in-memory chunking locally) because we need to calculate the `md5` hash of the data (which can only be done locally) in order to `dvc add` it properly. Maybe for DVC 3.0 though, we’ll see! And we still welcome any feature requests on our Github repo.
Thanks again!
---
### Author:  [@marius](https://discuss.dvc.org/u/marius)
#### Post date: [March 17, 2021, 1:07am UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/8 "2021-03-17T01:07:39Z")
> [@jorgeorpinel](#):
>
> BTW: the transfer doesn’t actually use any local disk space at all, as the chunks are stored locally in memory only.
Hmm I suppose then there’s something I’m not understanding about what was happening, will look more, but in any case the info here is very helpful, thanks!
---
### Author:  [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)
#### Post date: [March 17, 2021, 1:56am UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/9 "2021-03-17T01:56:06Z")
Well, it still needs to download **and upload** each chunk (into memory) so the transfer still gets bottlenecked locally 🙂
---
### Author:  [@marius](https://discuss.dvc.org/u/marius)
#### Post date: [March 17, 2021, 2:13am UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/10 "2021-03-17T02:13:52Z")
To clarify, “scratch” and “home” filesytems I’m referring to here are two different network filesystems both mounted on the same machine. As far as DVC sees, there are only file copies, no upload/download. It happens that scratch \<-\> scratch is fast scratch \<-\> home is slow, the bottleneck I think being home disk and/or network speed, so I don’t think its an issue that data passes through memory (which in this configuration to me seems unavoidable to copy from one filesystem to another).
---
### Author:  [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)
#### Post date: [March 17, 2021, 5:50am UTC](https://discuss.dvc.org/t/how-to-add-data-from-scratch-filesystem-directly-to-scratch-cache-dir/702/11 "2021-03-17T05:50:08Z")
Ah yes good point. I guess still in this case the chunked transfer with Python-level i/o implemented in DVC may still be significantly slower than a native file system call i.e. `cp`.