# Storage usage for large file versioning

**URL:** https://discuss.dvc.org/t/storage-usage-for-large-file-versioning/951
**Category:** Feature Requests
**Created:** [November 5, 2021, 9:21am UTC](https://discuss.dvc.org/t/storage-usage-for-large-file-versioning/951 "2021-11-05T09:21:37Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Hao](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Hao](https://discuss.dvc.org/u/Hao)
#### Post date: [November 5, 2021, 9:21am UTC](https://discuss.dvc.org/t/storage-usage-for-large-file-versioning/951/1 "2021-11-05T09:21:37Z")

</div>

Hi, several months ago, I asked a question about the mechanism DVC use to manage large file versioning.

The example given was for a 10 Go file managed by DVC, a new version of the file is committed with a very small change. What will be the storage impact on this commit?

The answer I got was that the storage usage will be doubled, so 20 Go used after the new version’s commit. Because DVC doesn’t compare changes inside a large file.

I understand there was a plan for DVC to manage changes in large file. So that in my example, instead of double the storage usage, only the changed part will be stored on top of the base version.

My questions are :

1. Is there a timeline to implement this new feature?
2. Does DVC compare file content while versioning small files?

---

<div class="post-metadata">

### Author: ![YanxiangGao](https://avatars.discourse-cdn.com/v4/letter/y/f04885/32.png) [@YanxiangGao](https://discuss.dvc.org/u/YanxiangGao)
#### Post date: [November 6, 2021, 1:59am UTC](https://discuss.dvc.org/t/storage-usage-for-large-file-versioning/951/2 "2021-11-06T01:59:23Z")

</div>

1. Is there a timeline to implement this new feature?  
The current plan is to start this in 2022 Q1.

2. Does DVC compare file content while versioning small files?  
For two files with the same contents, DVC will only save one copy for them.

---

<div class="post-metadata">

### Author: ![Hao](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@Hao](https://discuss.dvc.org/u/Hao)
#### Post date: [November 6, 2021, 9:05am UTC](https://discuss.dvc.org/t/storage-usage-for-large-file-versioning/951/3 "2021-11-06T09:05:49Z")

</div>

Thank you for your prompt response.

1. Is there any expected date/timeframe that this feature will become available to end users? What will be the impact to the already versionned files?
2. May I understand there is currently no content compare for small files? For a tiny change in a small file, a second copy will be made (as for large file).

---

<div class="post-metadata">

### Author: ![YanxiangGao](https://avatars.discourse-cdn.com/v4/letter/y/f04885/32.png) [@YanxiangGao](https://discuss.dvc.org/u/YanxiangGao)
#### Post date: [November 7, 2021, 2:07am UTC](https://discuss.dvc.org/t/storage-usage-for-large-file-versioning/951/4 "2021-11-07T02:07:00Z")

</div>

1. Is there any expected date/timeframe that this feature will become available to end users? What will be the impact to the already versionned files?  
Depending on when it would be started, still lots of work to be done on the underlying levels. We will try to make it compatible with the old version of caches, but details are only available after we began working on it.

2. May I understand there is currently no content compare for small files? For a tiny change in a small file, a second copy will be made (as for large file).  
Yes, we just save every one of them independently.

---

<div class="post-metadata">

### Author: ![YanxiangGao](https://avatars.discourse-cdn.com/v4/letter/y/f04885/32.png) [@YanxiangGao](https://discuss.dvc.org/u/YanxiangGao)
#### Post date: [November 12, 2021, 8:23am UTC](https://discuss.dvc.org/t/storage-usage-for-large-file-versioning/951/5 "2021-11-12T08:23:20Z")

</div>

Here is an issue for this, from it you can know the latest update on this question.

> <https://github.com/iterative/dvc/issues/829#issuecomment-966368458>
>
> As @shcheklein suggested, we should consider splitting data into small blocks to… track data changes more efficiently. Example: giant file that has one line appended to it.
