# Can the command \`dvc add\` just add new and changed files?

**URL:** <https://discuss.dvc.org/t/can-the-command-dvc-add-just-add-new-and-changed-files/262>\
**Category:** Questions\
**Created:** [December 1, 2019, 11:38am UTC](https://discuss.dvc.org/t/can-the-command-dvc-add-just-add-new-and-changed-files/262 "2019-12-01T11:38:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sven](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/sven/32/51_2.png) [@Sven](https://discuss.dvc.org/u/Sven)\
**Post date:** [December 1, 2019, 11:38am UTC](https://discuss.dvc.org/t/can-the-command-dvc-add-just-add-new-and-changed-files/262/1 "2019-12-01T11:38:58Z")

</div>

I have just started playing with dvc to investigate whether it can help me adding version control to large sets of images used for deep learning.

I made my own _small_ toy example to try to understand how dvc works.

I have a data folder with 10 images of approx 10 MB each, in total ~100 MB

When I run “dvc add data” I can see that the dvc cache grows with ~100 MB

I now replace one of the images in my data folder with another one, i.e. remove one 10 MB image and add another 10 MB image. The total size is still 100 MB

When I run “dvc add data” again to track the changes, the dvc cache grows by another 100 MB! I was thinking that dvc would automatically add the difference between the data sets, but that does not seems to be the case? Does it _copy_ all the content for each “dvc add” command?

If yes, is there any practically convenient way to make dvc detect and store the _difference_ between versions of data sets?

Thanks,  
Sven

---

<div class="post-metadata">

**Author:** ![kupruser](https://avatars.discourse-cdn.com/v4/letter/k/e95f7d/32.png) [@kupruser](https://discuss.dvc.org/u/kupruser)\
**Post date:** [December 1, 2019, 11:58am UTC](https://discuss.dvc.org/t/can-the-command-dvc-add-just-add-new-and-changed-files/262/2 "2019-12-01T11:58:20Z")

</div>

Hi @Sven !

That is odd, how do you check the size of the cache dir? Dvc should already store all old files plus new file, so it should be 110MB total in the cache, that is a core feature for us.

---

<div class="post-metadata">

**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\
**Post date:** [December 1, 2019, 6:24pm UTC](https://discuss.dvc.org/t/can-the-command-dvc-add-just-add-new-and-changed-files/262/3 "2019-12-01T18:24:48Z")

</div>

Hi. Here’s a test I ran with 3 5mb files, which seems correct (no file duplication):

```auto
$ dvc version
DVC version: 0.71.0
Python version: 3.7.5
Platform: Darwin-19.0.0-x86_64-i386-64bit
Binary: False
Package: brew
Cache: reflink - True, hardlink - True, symlink - True

$ mkdir test && cd test
$ dvc init --no-scm
...
$ mkdir data
$ head -c 5242880 /dev/urandom > data/5mbfile1
$ head -c 5242880 /dev/urandom > data/5mbfile2
$ ls -lAFh
drwxr-xr-x 5 usr staff 160B Dec 1 12:07 .dvc/
drwxr-xr-x 4 usr staff 128B Dec 1 12:11 data/
$ ls -lAFh data
-rw-r--r-- 1 usr staff 5.0M Dec 1 12:11 5mbfile1
-rw-r--r-- 1 usr staff 5.0M Dec 1 12:11 5mbfile2
$ dvc add data
100% Add ...
$ du -sh .dvc/cache
 10M	.dvc/cache
$ rm -f data/5mbfile2
$ head -c 5242880 /dev/urandom > data/5mbfile3
$ ls -lAFh data
-rw-r--r-- 1 usr staff 5.0M Dec 1 12:11 5mbfile1
-rw-r--r-- 1 usr staff 5.0M Dec 1 12:11 5mbfile3
$ dvc status
data.dvc:                                                                                                                                    
	changed outs:
		modified: data
$ dvc add data
WARNING: Output 'data' of 'data.dvc' changed because it is 'modified'                                                                        
100% Add ...
$ du -sh .dvc/cache
 15M	.dvc/cache

```

Maybe you can help us reproduce the problem with instructions such as these? Thanks

---

<div class="post-metadata">

**Author:** ![Sven](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/sven/32/51_2.png) [@Sven](https://discuss.dvc.org/u/Sven)\
**Post date:** [December 3, 2019, 2:19pm UTC](https://discuss.dvc.org/t/can-the-command-dvc-add-just-add-new-and-changed-files/262/4 "2019-12-03T14:19:34Z")

</div>

Thanks for your replies and help!

Having tested the commands supplied by jorgeorpinel, it see that “du -sh” does not report the same folder size as right clicking on it and selecting “properties” in Windows explorer.

If I take “du -sh” as the truth, everything works as expected. I guess that Windows does not does not compute the disk size on disk correctly?

---

<div class="post-metadata">

**Author:** ![kupruser](https://avatars.discourse-cdn.com/v4/letter/k/e95f7d/32.png) [@kupruser](https://discuss.dvc.org/u/kupruser)\
**Post date:** [December 3, 2019, 6:20pm UTC](https://discuss.dvc.org/t/can-the-command-dvc-add-just-add-new-and-changed-files/262/5 "2019-12-03T18:20:44Z")

</div>

We do indeed trust `du -sh` a lot 🙂 Not sure how that is implemented in Windows explorer. Our team member also noted that it might be caused by it not quite understanding the hardlinks we use for some optimisation and counting them twice. Sounds reasonable to me as well 🙂