# Shared cache on NFS is slow

**URL:** <https://discuss.dvc.org/t/shared-cache-on-nfs-is-slow/892>\
**Category:** Questions\
**Created:** [September 16, 2021, 2:29am UTC](https://discuss.dvc.org/t/shared-cache-on-nfs-is-slow/892 "2021-09-16T02:29:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmiller](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jmiller](https://discuss.dvc.org/u/jmiller)\
**Post date:** [September 16, 2021, 2:29am UTC](https://discuss.dvc.org/t/shared-cache-on-nfs-is-slow/892/1 "2021-09-16T02:29:49Z")

</div>

I have multiple users who each can launch their own AWS instance for data analysis, then shut it down when they are done. Rather than having to rebuild the dvc cache on each of these instances when they startup, we want to have a persistent shared cache.

So I’ve created a shared DVC cache on EFS (AWS’s version of NFS) with symlinks in the workspace, but the problem is that ‘dvc fetch’ and ‘dvc checkout’ are now quite slow for certain branches that have 1000s of files tracked by DVC. First, for a ‘dvc fetch’, even if no downloads need to be done from the remote, just “Querying cache…” can take up to 8 minutes. Then ‘dvc checkout’ can take 6 more minutes. Presumably this has to do with NFS communication overhead to list files in the cache (I see the progress bar saying things like “110 files/sec”).

Is there any way to speed up a shared cache stored on NFS, especially when you have 1000s of tracked files?

---

<div class="post-metadata">

**Author:** ![YanxiangGao](https://avatars.discourse-cdn.com/v4/letter/y/f04885/32.png) [@YanxiangGao](https://discuss.dvc.org/u/YanxiangGao)\
**Post date:** [September 16, 2021, 2:24pm UTC](https://discuss.dvc.org/t/shared-cache-on-nfs-is-slow/892/2 "2021-09-16T14:24:47Z")

</div>

Could you please get the cprofile dump of your command, It can be get via  
adding a `--cprofile --cprofile-dump my.prof` option in your command.

---

<div class="post-metadata">

**Author:** ![jmiller](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jmiller](https://discuss.dvc.org/u/jmiller)\
**Post date:** [September 20, 2021, 4:24pm UTC](https://discuss.dvc.org/t/shared-cache-on-nfs-is-slow/892/3 "2021-09-20T16:24:13Z")

</div>

Is any private information possibly included in these dump files (passwords, AWS keys, file or variable contents, etc)? I am a little hesitant to post the full files because of that.

But I analyzed the files with snakeviz and can tell you that for “dvc fetch” the vast majority of time was spent in `hashes_exist > ... > ~:0(<built-in method posix.stat>)`.

 ![Screen Shot 2021-09-20 at 9.19.12 AM](https://canada1.discourse-cdn.com/flex035/uploads/dataversioncontrol/original/1X/04fff4daec7df3174fd060b9cea809e1a7ceba2e.png)

 ![Screen Shot 2021-09-20 at 9.50.53 AM](https://canada1.discourse-cdn.com/flex035/uploads/dataversioncontrol/original/1X/b485a11b1e4d99840d0b684308d3f702df3e3de0.png)

Same for “dvc checkout” – it all comes down to posix.stat.

 ![Screen Shot 2021-09-20 at 9.22.29 AM](https://canada1.discourse-cdn.com/flex035/uploads/dataversioncontrol/original/1X/6fdc50733f6e0bea5d555410c87251f69d9c7be9.png)

---

<div class="post-metadata">

**Author:** ![YanxiangGao](https://avatars.discourse-cdn.com/v4/letter/y/f04885/32.png) [@YanxiangGao](https://discuss.dvc.org/u/YanxiangGao)\
**Post date:** [September 21, 2021, 4:05am UTC](https://discuss.dvc.org/t/shared-cache-on-nfs-is-slow/892/4 "2021-09-21T04:05:16Z")

</div>

Guess it is related to [https://github.com/iterative/dvc/issues/5562](https://github.com/iterative/dvc/issues/5562) this one?

---

<div class="post-metadata">

**Author:** ![jmiller](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jmiller](https://discuss.dvc.org/u/jmiller)\
**Post date:** [September 22, 2021, 10:17pm UTC](https://discuss.dvc.org/t/shared-cache-on-nfs-is-slow/892/5 "2021-09-22T22:17:07Z")

</div>

Yes, looks like it. I’ll look forward to any improvements that could be made on the DVC side.

For those using Amazon’s EFS, I recommend using General Purpose mode (not max I/O), which has lower latency and thus appears to speed up the stat() calls.